Saturday, March 19, 2011

What skills do you need to become good data miner?

This is the question I am often asked.

In order to become good data mining practitioner one needs to understand statistical concepts and basic principles of knowledge induction. Knowing inferential stats, t-tests, analysis of variance, regressions, etc.. is important, and this is your "bread and butter" knowledge. Then, you need to know some academic or commercial software, since you won't be doing logistic regression - for example - on a piece of paper.

At this point, you at the mercy of educational institution. If they have access to software that large organizations in your local markets are using - well, then your employability will be far higher - than if they open source academic software.

In the area where I live - there is one university whose statistic graduates are rarely seen in big companies. And it is no surprise - people who run this university have attitude of "why pay a cent for software "x" when I get software "y" for free? Well, if market demand is for software "x" - that should be reason good enough.

So, you know the stats and you know some statistical software - is that it? Sure is not. Few times I had statisticians with master degree and phd’s phoning me to ask me "where do I start with solving my business problem? So, knowing methodologies and processes is vitally important. No predictive modeling type of project will ever start with a predictive model. In fact no data mining project will ever start with data or with statistical/data mining software. So, methodology is not what you learn in most stat classes! Right?

Therefore, is knowing statistical methods, some statistical software and methodologies all you need to know to become good analyst/data miner? Not quite! How about knowledge of applications that you would use your statistical skills? If you a building credit-card fraud detection model, as oppose to policy lapse prediction model - basic principles of model building are the same, but modeling nuances are not. I often say that good data miner understands that everything that happens in data preparation phase, as well as in modeling phase is a function of business problem and business objectives. So, one can imagine that there are big differences between fraud in the credit-card space as oppose to insurance policies lapses. And those differences will be translated into how you are going to prepare your data and how will you build your model.

Therefore, knowing applications is very important. And, regardless of in which industry you are - some applications you will encounter more often than the others. Segmentations, cross-selling, retention/churn, customer value and profitability are particularly common - so, every practitioner should be familiar with at least those.

So, is that all? Nope.

How about knowledge of best practices? Instead of relying on your intuition and gut-feeling when confronted with specific modeling puzzle, wouldn't be helpful to know what others have done when confronted with the same problem, and how successful they were. That is where knowing best practices come handy. And if solution to your modeling problem is not in any best practices manual - then relying on your intuition and creativity becomes essential.

So, let's imagine you have all these skills, and you are required to present your results to the board of your multinational client - which is some Chinese company. And it gets worse! You need to present these results in Mandarin. And you can't find anyone to translate it. And this means you will not be able to convince the board that you will dramatically reduce their business problem. And, all that means that with all your skills - your project has FAILED!

What am I trying to say here? You got to have ability to communicate yo your business sponsors - in a way that they can understand you, so that you can convince them that what you have done will have positive business effects. So you have to be able to switch between two different languages - language of techies and language of business folk. And this is not the easy task!

I have seen good modelers blowing it all up by putting their business audiences to sleep, and I saw modelers having huge success with mediocre modeling results, just by virtue of by being able to talk the language of their business audiences.

So, now you can see why data miner is rare creature. And hopefully, this article can help you in identifying your own areas of improvement.

Good luck and send me comment!


  1. Nice blog Goran. Reading through the rest of it now. Interesting!

  2. Great article! You basically touched upon different skills for an ideal data mining analyst. In my dissertation, I investigated the role of human expertise and the value of data mining technology. The results showed that an ideal data analyst ought to have a good level of expertise in technology, statistical modeling and analysis, knowledge of data, domain knowledge, and communication. Although an analyst may not have all five skills, a data mining team should demonstrate a high level of expertise in those skills.

  3. Hi,
    Good to see your articles,

    here also i will share some similar kind of business services.
    We take great care in ensuring your lists are as accurate and updated as possible. We ensure that our accuracy rates are above the industry standard. USA B2B Email Database

    USA Hospitality Email Lists
    USA Investors Email List
    USA CMO Email Lists

  4. A blog of the analytics and the data mining with the help of which you can enjoy the new links in the better way. You can come to this homepage and can enjoy the new skills in this place. You can also find more pages of information from this area.

  5. Data mining! Specialists are responsible for designing various data analysis services to mine for business process information. ... The attached document is a job description template for a Data Mining Specialist and its also use for
    b2b leads!