The Association Rules algorithm focuses on itemsets, which you can think of as a bundle of “things” (attributes) which frequently go together. For example, a popular itemset might be {Peanut Butter, Jelly, Bread}. Itemsets don’t have to be related to shopping, however. A perfectly valid itemset might be {Red Sox Fan, Cretin, Convict}: We’re just creating groups of attributes which go together.

Given many itemsets, the Association Rules algorithm discovers most frequently occurring ones. Then, it creates association rules using those frequently occurring itemsets.  Here’s what a rule looks like:

{Peanut Butter = Existing , Jelly = Existing => Bread = Existing}

If we see Peanut Butter and Jelly in someone’s shopping cart, they are likely to buy Bread, as well.

The concepts of support, probability, and importance are used to tell us how many times we see an item or itemset, the confidence we have in the rule, and the “lift” we might expect for a rule.

If you’d like to understand more about these concepts, read this thread by Jamie Maclennan.

Here’s a basic query to return information from our data mining model:

SELECT * FROM OpenQuery (SSAS,’SELECT NODE_CAPTION as [Rule], NODE_SUPPORT AS [Support], MSOLAP_NODE_SCORE AS [Importance],

NODE_PROBABILITY AS [Probability] from

[Association].content WHERE NODE_TYPE=8′)

Executing this puppy will return all our rules and their support,  probability & importance values. We’re looking for rules with relatively high importance (> 0) which indicate that the existence of peanut butter and jelly will impact bread buying positively:

If you open the image above, you’ll see that each rule looks pretty much like a “formula”. To make these formulae useful, we’re going to need to parse them into individual fields that be used as labels on an axis. To do that, we write lots of kludgy string parsing code:

SELECT

    [Rule],

    — Cast NTEXT to nvarchar then determine number of items in itemset by keying on number of = symbols before the -> characters

    — Divide by 2 since DATALENGTH against NVARCHAR will be twice the string length

    (DATALENGTH(LEFT(CAST([rule] as nvarchar(300)),CHARINDEX(‘->’,cast([RULE] as nvarchar(300)))))- DATALENGTH (REPLACE(LEFT(CAST([rule] as NVARCHAR(300)),CHARINDEX(‘->’,[RULE])),’=',”)))/2   as [ItemSet Size],

    [Support],

    [Importance],

    [Probability],

    — Grab first item in itemset

    LEFT(CAST([rule] as nvarchar(300)), CHARINDEX(‘=’, CAST([rule] as nvarchar(300))) -1) as [Basket Item 1],

    — If two items exist in itemset, grab second using kludgy string parsing

    CASE (DATALENGTH(LEFT(CAST([rule] as nvarchar(300)),CHARINDEX(‘->’,cast([RULE] as nvarchar(300)))))- DATALENGTH (REPLACE(LEFT(CAST([rule] as NVARCHAR(300)),CHARINDEX(‘->’,[RULE])),’=',”)))/2

        WHEN 2 Then SUBSTRING(CAST([rule] as NVARCHAR(300)),CHARINDEX(‘,’,CAST([rule] as nvarchar(300)))+2, ((CHARINDEX(‘->’,cast([RULE] as nvarchar(300)))-12) – (CHARINDEX(‘,’,CAST([rule] as nvarchar(300)))+2) ))

    END as [Basket Item 2],

    — Recommended Item

    REPLACE(RIGHT(CAST([rule] as NVARCHAR(300)), (DATALENGTH(CAST([rule] as NVARCHAR(300)))/2) – (CHARINDEX(‘->’,cast([RULE] as nvarchar(300)))+2)),’ = Existing’,”) as [Recommended Item]

FROM

OpenQUery (SSAS,’SELECT NODE_CAPTION as [Rule], NODE_SUPPORT AS [Support], MSOLAP_NODE_SCORE AS [Importance],

NODE_PROBABILITY AS [Probability] from

[Association].content WHERE NODE_TYPE=8′)

Writing the SQL above was by far the ugliest bit. On to the results:

The resultset above adds “Basket Item 1”, “Basket Item 2”, and “Recommended Item” fields by parsing the the rule column.

Now for the fun stuff. First, let’s look at our rules by Importance and Probability:

Next, lets look at the actual products which are being recommended:

Support is being used you determine our bubble size, so we can see below that Mountain Bottle cages are recommended by 5 different rules:

By highlighting a specific item (product), we can see that while we generated lots of rules which recommend it, those rules aren’t very important – I’d probably ignore these.

Finally, let’s see which items  tend to go together in the same basket. Here we can see that the Mountain Bottle Cage gets grouped with a lot of other items.

One will generally use the clustering algorithm to segment “things” (customers into “customer clusters”, for example) based on similarities between attributes that describe the “things” in question.

The Adventure Works sample cube has a ready-made data mining structure which contains a customer-focused clustering model. It’s called Customer Clusters and takes the following inputs:

• Commute Distance
• Education
• Gender
• Home Owner
• Marital Status
• Number of Cars Owned
• Number of Children at Home
• Occupation
• Total Children
• Yearly Income

Rather than recreate the wheel, I’ll use the sample.

The first thing we’ll probably want to understand is how different combinations of attributes and attribute values combine to create distinct clusters of customers – otherwise known as a “cluster profile”. SQL Server Analysis Services has a quick but unsupported system stored procedure named GetClusterProfiles() which will give us this information, and since it’s the weekend, I don’t care if it’s unsupported. I’m going to execute this sproc against my data mining model:

Click and view the image above as there’s a lot of good information to be found. The columns 001-010 represent 10 distinct clusters of customers that were discovered based on the data the model was trained on. Attribute Name and Attribute Value show the inputs and potential input values that were found in the data. For each attribute name/value pair, you’ll see a support and probability value. Support represents the number of cases (rows) we found that fit the criteria in question. Probability tells us the probability a particular attribute name/value pair will show up in the customer cluster in question

For example in Customer Cluster 001, there is a ~ 42% chance the customer lives less than a mile away from work, about a 1% chance the person lives 10+ miles away from work, and a ~25% chance they live within 1-2 miles, etc.

As an aside, it’s normal to actual rename clusters from 001, 002, 003 to something more meaningful like “Soccer Moms” , “Yuppies” or “Hipsters” based on our read of the primary drivers of each cluster. I haven’t bothered, so we’ll refer to our clusters as 001-010.

As I mentioned, I initially wanted to use GetClusterProfile()’s output as my data source, but found that I couldn’t use OPENQUERY and the Microsoft MSOLAP provider to execute it. I’m guessing that the fact we return an unknown number of columns (based on the number of clusters we find) could be causing the problem.

Instead, I had to use multiple DMX statements “UNIONed” together. Each DMX SELECT returns cluster attribute name/value pairs for a single attribute. Here’s one for Commute Distance:

There’s something important here to note, by the way. Notice the WHERE clause where I filter out any attributes with less support than 500 cases. Choosing 500 was an arbitrary decision on my part, but it impacted my results in a big way. In the output above, there are now only 3 Commute Distance nodes returned for customer cluster 001, while there were 5 when I executed GetClusterProfile() in the first screenshot. Is this good or bad? Hard to say – there were only 35 cases supporting the “1% probability of living 10+ miles away” node in cluster 001. Is 35 statistically significant enough to include?

(By the way, after the whole discussion around SUPPORT vis-à-vis “Customer Commute” I managed to completely FORGET to add it to the final query above! Since this is just an example I decided not to go back and correct my oversight, however – don’t be surprised when you don’t see the Commute Distance attribute in any of the following vizes.)

After dropping our final SELECT statement into a Tableau “Custom SQL” data connection, we’re ready to build a Viz:

• The Node_Name (The name of the cluster: 001, 002, etc.) pill lands on Columns
• Attribte_Name and the Probability measure get dropped in Rows
• Attrbute_Value lives in the Color shelf

You can now easily see how each attribute value drives the composition of different customer clusters (001-010, on columns).

I found there were mixed text and number values in the Attribute_Value column, and this was painful. For example, I had to deal with values like “Married” and “Single” (Marital Status) and .056365 (Number of Cars Owned in cluster 010) in the same field. I used Groups to simplify things a bit, but I’d probably massage the data model / query a bit more if I were doing this “for real” rather than “for fun”.

After we understand how customer clusters are composed, we can really have some fun.

The Adventure Works sample SSAS database contains a second cube called “Mined Customer”. Mined Customers combines two data mining models (Customer Clusters and an Association Rules model for market basket analysis) with lots of other metrics like sales, profit. The mash up of our “default” metrics and dimensions with the “mining insights” gives us some exciting possibilities for visualization.

In the viz below, we’re able to look at average sales and profit margin by our ten customer clusters. One can see that Cluster 5 drives the highest profit margin with a relatively large number of customers:

And the fun continues! By dropping one of the “demographic inputs” on the Level of Detail shelf, we can begin exploring within individual clusters. I’m going to drill in by Occupation.

It now becomes apparent that skilled manual workers in Cluster 6 are more profitable than Cluster 5 as a whole. By putting together my Quick Filters and Demographic fields on the Level of Detail shelf I also have the ability to completely explore the sales and profitability of my customer clusters.

More Cowbell

And I have an admission to make: I’m obsessed. Since Gartner released its 2011 Magic Quadrant for Business Intelligence Platforms report a few weeks ago, I’ve had MQ on the brain. I’ve been thinking about MQs non-stop – it’s my cowbell. In fact, I got a fever, and the only prescription…is more cowbell!!”

So, I decided I’d take the previous viz and turn it into a MQ. I essentially zoomed in a tad, and dropped in four annotation areas with no labels and played around with background shading. Voila, more cowbell!

Next time, Association Rules – otherwise known as market basket analysis

One thing that has occurred to me as a potential bump in the road towards wider adoption  is visualization. Microsoft offers some sample (and admittedly long-in-the-tooth) data mining visualizations but perhaps users still find it too hard to get their newly won insights out of SQL and/or Excel and present same to rest of the world? Maybe users want more control over how the visualizations look and feel and how one interacts with the results?

To that end, I thought it would be fun to work with a product from Microsoft gold certified partner, Tableau Software. The folks at Tableau make a pretty awesome piece of BI goodness named Tableau Desktop that has really, really nice visualization capabilities. Tableau Desktop is a bit like my old favorite ProClarity, except that it can consume data from SSAS to SQL to Teradata to Netezza to PowerPivot and back to text files. Did I mention the tool has awesome visualization capabilities?

What I’m going to do over the next few weeks or as time allows is to drop Tableau on top of SQL Server Analysis Services data mining output. I’ll try to come up with meaningful examples of how one might visualize the output of all the major data mining algorithms SSAS supports. Sounds fair?

Time Series

I’ll start with the Time Series algorithm, which we’ll use for predicting values based on numbers tied to slices of time in the past. One essentially feeds a Time Series model a bunch of numbers which are each associated with a timestamp, and SSAS can come back to you with predicted values for other time periods.

In PerformacePoint 2007, we took advantage of this functionality directly and had a really cool report type which one could nest in a dashboard. Unfortunately, the report type leveraged the deprecated Office Web Components, so we don’t have it PPS 2010. In honor of my lost friend, we start with the Time Series:

I’m going to assume you know something about Microsoft data mining. If you don’t, I advise you to spend some time with the fine tutorial and books online. As much as I can, I’ll actually base my work off these walkthroughs and examples in the sample AdventureWorks cube.

So, we’ll start off in BIDS with a simple DMX query which predicts the next 6 periods (months) of sales based on history:

I’m re-using the “Forecast” model that is already waiting for us inside the SQL Serve Analysis Services sample cube AdventureWorks DW 2008R2. The query will return the model of item we’re selling, the region it’s being sold in and 6 months worth of predictions.

The “as-is” query has a problem in that the results come nested as distinct datasets.  This is pretty hard to use :

We’ll fix this annoyance by leaning on the DMX FLATTEN statement. One can flip from ‘Designer” to “SQL” view and modify the query itself:

SELECT FLATTENED

  [Forecasting].[Model Region],

  (PredictTimeSeries([Forecasting].[Amount],6)) as [Amount],

   True as [Predicted]

From

  [Forecasting]

Note how I’ve also added a hard-coded field – “Predicted”; I’ve set the value to “True” to indicate that the values in question are predictions, not historical values.

The look that we’re ultimately going for in our visualization is a single, unbroken line or bar chart which gradually transitions from plotting historical values to predicted values. The bit above gets us our “in the future” numbers.

Getting the historical numbers is pretty easy because there happens to be a SQL Server view which provides them to the data mining algorithm for training purposes. The view is named vTimeSeries, and I’ll just re-purpose it:

SELECT ModelRegion,

   TimeIndex,

   Amount,

   0 as [Predicted] –-‘False’: Historical values

FROM vTimeSeries

How do we merge these two distinct sets of data? UNION:

SELECT * FROM OpenQuery (SSAS,’SELECT FLATTENED

  [Forecasting].[Model Region],

  (PredictTimeSeries([Forecasting].[Amount],6)) as [Amount],

   True as [Predicted]

From

  [Forecasting]‘)

UNION ALL

SELECT

   ModelRegion,

   TimeIndex,

   Amount,

   0 as [Predicted]

FROM vTimeSeries

Note that we use OpenQuery to make this all work – we therefore have a Linked SQL Server Analysis Server plugged into our SQL Server. I’ve named it SSAS, and here’s the property dialog:

Once the linked server is there, we can test our query:

Now that the query is done, the real fun can begin. In a new Tableau workbook, we’ll need to create a new Data Connection pointing at SQL Server and using the Custom SQL option. Paste the query above into the Custom SQL dialog, and you’re pretty much ready roll.

• Drop your Time dimension on Columns
• Drop ModelRegion and Amount on Rows
• Place the “new” Predicted dimension on the Color shelf.

If you click the image above, you’ll see that the Predicted field drives the historical/predicted value transition via a color change.

I also added a few Calculated Fields to break out “Region” and “Model” into two distinct dimensions and added each as a filter. For kicks, I changed my shapes from bars to prediction-or-not influenced squares and circles:

That’s it!

While testing that I could return results from DMX via OpenQuery, I ran a basic DMX statement to return results from a time series model:

SELECT * FROM OpenQuery (SSAS,'SELECT FLATTENED
  [Forecasting].[Model Region],
  (PredictTimeSeries([Forecasting].[Amount],3)) as [Amount],
   True as [Predicted]
FFrom
  [Forecasting]')

Doing so returned this error message:

   OLE DB provider "MSOLAP" for linked server "SSAS" returned message "Error (Data mining): The '\\?\C:\Program Files\Microsoft SQL Server\MSAS10_50.MSSQLSERVER\OLAP\Data\Adventure Works DW 2008R2.0.db\Forecasting.0.dms\Forecasting.0.dmm\0.Forecasting.cnt.bin' persisted file cannot be found.".
Msg 7321, Level 16, State 2, Line 1
An error occurred while preparing the query "SELECT FLATTENED
  [Forecasting].[Model Region],
  (PredictTimeSeries([Forecasting].[Amount],3)) as [Amount],
   True as [Predicted]
From
  [Forecasting]" for execution against OLE DB provider "MSOLAP" for linked server "SSAS".

Problem? Stupid user trick – I hadn’t processed the model.  I was surprised to find that this error message couldn’t be found via Bing/Google, however. So here’s a little entry to save someone else some time.

Cheers.

Since SharePoint/PerformancePoint 2010 now includes a decomp tree, I generally don�]]> http://performancepointblog.com/2010/11/creating-a-data-bound-drillable-heat-map-with-silverlight/feed/ 5 http://performancepointblog.com/2010/08/re-creating-the-powerpivot-management-dashboard%e2%80%99s-%e2%80%9csliding-bubble-chart%e2%80%9d/ http://performancepointblog.com/2010/08/re-creating-the-powerpivot-management-dashboard%e2%80%99s-%e2%80%9csliding-bubble-chart%e2%80%9d/#comments Thu, 05 Aug 2010 19:41:36 +0000 Russell http://performancepointblog.com/?p=254 http://performancepointblog.com/2010/08/re-creating-the-powerpivot-management-dashboard%e2%80%99s-%e2%80%9csliding-bubble-chart%e2%80%9d/feed/ 9 http://performancepointblog.com/2010/06/creating-pivot-collections-via-reporting-services-sql-analysis-services-challenges-and-solutions/ http://performancepointblog.com/2010/06/creating-pivot-collections-via-reporting-services-sql-analysis-services-challenges-and-solutions/#comments Wed, 30 Jun 2010 12:35:32 +0000 Russell http://performancepointblog.com/?p=178 http://performancepointblog.com/2010/06/creating-pivot-collections-via-reporting-services-sql-analysis-services-challenges-and-solutions/feed/ 0 http://performancepointblog.com/2010/01/powerpivot-dax-and-semi-additive-measures/ http://performancepointblog.com/2010/01/powerpivot-dax-and-semi-additive-measures/#comments Thu, 21 Jan 2010 16:45:29 +0000 Russell http://performancepointblog.com/?p=149 http://performancepointblog.com/2010/01/powerpivot-dax-and-semi-additive-measures/feed/ 1 http://performancepointblog.com/2009/11/creating-useful-powerpivot-data-models-for-public-consumption-via-reporting-services/ http://performancepointblog.com/2009/11/creating-useful-powerpivot-data-models-for-public-consumption-via-reporting-services/#comments Wed, 25 Nov 2009 20:13:29 +0000 Russell http://performancepointblog.com/?p=124 http://performancepointblog.com/2009/11/creating-useful-powerpivot-data-models-for-public-consumption-via-reporting-services/feed/ 4 http://performancepointblog.com/2009/11/ctp3-setup-error-could-not-load-file-or-assembly-microsoft-analysisservices-sharepoint-integration/ http://performancepointblog.com/2009/11/ctp3-setup-error-could-not-load-file-or-assembly-microsoft-analysisservices-sharepoint-integration/#comments Wed, 18 Nov 2009 21:38:17 +0000 Russell http://performancepointblog.com/?p=98 http://performancepointblog.com/2009/11/ctp3-setup-error-could-not-load-file-or-assembly-microsoft-analysisservices-sharepoint-integration/feed/ 0