Microsoft has been a great place to work, with many (many, many, many) people smarter than I am to lean on and help me grow. I wouldn’t be where I am now without my blue badge.

However, I’ve managed to stumble into what I think will be another one of those “damn lucky” opportunities. I’m going to work for Tableau Software.

Over last year’s Christmas break, I played with a sampling of BI tools from “the new kids on the block”.  In fact, I wrote about some of my experiences here. Working with Tableau Desktop, however…it shifted my frame of reference around what business intelligence can be both quickly and dramatically (and for me, the beauty of Tableau is that I can continue to use it with products that I know and love like SQL Server ant SharePoint).

I’ll be joining Tableau as a Business Consultant in late-July, and I’m excited as hell.

I’m frankly not sure what will happen to performancepointblog.com: I’ll definitely keep it up. However, I tend to blog about what I do, so I suspect a new data visualization blog with a Tableau focus will make a debut sometime in the near future.

“Thanks!”, Microsoft…and “Hello”, Tableau!

Wesley Backelant fired off an early morning tweet on the newly-released Microsoft Analytics add-in for Twitter, and I gotta say that I’m pretty excited. Twitter is my social media tool of choice, and I’ve been playing around with various .NET twitter-related libraries out on codeplex in an effort to do many of things this add-in does out of the box.

Essentially the add-in is made up of:

• Logic to search twitter for users, keywords, and hashtags.
• A PowerPivot data model to store search results
• A set of Excel worksheets used to report on topics (keywords), people (users), sentiment (tone) and see individual tweets themselves

….and source code. Glorious, beautiful source code. The final worksheet in the add-in contains all the code you need to rebuilt this solution yourself. Bless you Microsoft & Extended Results: I haven’t giggled like a little girl at 6AM in quite a while.

A quick jump start:

First,  download it:

http://www.microsoft.com/download/en/details.aspx?id=26213

Here’s a Wiki for late-breaking issues with the add-in:

http://social.technet.microsoft.com/wiki/contents/articles/3296.aspx

After installing the add-in, you’ll want to insert keywords, hashtags, and people into the search window. I, of course, search for myself:

The add-in’s code takes over to do the dirty work with Twitter, and you’ll see:

…and:

You’ll  be prompted to do a manual data refresh both in the PowerPivot environment and  in “Excel proper”, and you’re ready to rock and roll.

Let’s run through the worksheets you can play with step-by-step, shall we?

Tone Dictionary

The Tone Dictionary is a handy “configurator” which lets you define what words should drive positive and negative sentiment on the “Tone” worksheet that we’ll look at later. I rather enjoyed reading some of the negative terms like “terrible”, “sucks” and “worthless”. Well done! All of these terms can be edited:

Topics

The Topics worksheet is your overview data – it includes some basic information like total tweets, retweets, mentions, and more:

There is an excellent array of pre-configured slicers to let you filter by search query, time of day, tone, users, mentions, and hashtags.

I give Microsoft and Extended Results big props for adding some nice time intelligence to this worksheet. I’ve filtered the results below so that you can see when I do the majority of my tweeting – before work, with a little bit of time spent after 5:

On an unsliced workbook you can immediately see that Wesley got a lot of @mentions, while I did a lot of tweeting against the #businessinteligence and #msbi hastags (two of my favorite haunts):

People

On the People worksheet you have the same set of slicers to work with, and the ability to see top tweeters, mentions, retweeters, and mentioners.

You can also spot who is tweeting “positively” and “negatively” based on the tone dictionary:

It looks like Cecillio is having a bad day over on #businessintelligence! Want to know why? Just go over to the Details worksheet and find the tweet in question:

Once I did my filtering, I found that he had just used an unhappy emoticon – he didn’t drop a “sucks”, “worthless”, or “terrible”, which left me a bit disappointed:

#WorkModeOn –>; #BusinessIntelligence de data desde el 2001 al 2011 WTF Cubo y mas Cubo hoy   #SQLServer2008

Tone

Finally, there’s the Tone worksheet – you’ll use it do determine how folks are feeling about people, topics, and anything else:

You can see that something “good” happened on the #businessintelligence hastag on 6/9, and that yours truly has been pretty neutral the last few days.

Wrap up

This is good stuff. It is eminently configurable with the ability to add additional slicers, leverage DAX,  and use calculated measures in PowerPivot. Including source code so you can rip down and rebuild this yourself? Genius! Download, use and love this thing!

If you’re interested in a little bit of “out of the box Mobile BI” and don’t want to buy 3rd party products, you can always view your published Excel Services & PowerPivot workbooks on the iPad.

Designing and/or re-factoring your reports for the device does take a bit of thought and planning, however. This post is a brain dump on things you should consider with some hints and tips. Most of what I’m going to give you falls into the “no duh” category, but I think you’ll pick up some useful nuggets.

Size Matters

Truth. Your biggest challenge will be building a report which relates a meaningful amount of information in the limited amount of screen real estate you’ll have to work with. You thought it was tough artfully cramming things into a PPS dashboard? Meh! Take the training wheels off!

Develop and Test Small

You may have an iPad to begin with, and even if you do it’s a pain in the rear to constantly refresh your changes on the device to see how things are looking. Develop and test as much as you can on your desktop/laptop.

Install Safari on your desktop and configure it to act like Safari-on-iPad:

1. Open Safari’s Options dialog.
2. Click Advanced on the Options dialog’s toolbar.
3. Select Show Develop menu in the menu bar.
4. if you want to go all out, click Security on the Options dialog, and de-select Enable plug-ins and Enable Java. This isn’t strictly necessary, but the items don’t work by default on some iOS Safari installations as far as I know.
5. Close the Options dialog. If the Menu Bar isn’t showing in Safari (File, Edit View, History, Bookmarks, etc.) Choose the Options drop down, and de-select Hide Menu Bar.
6. On the Options drop-down, select Hide Bookmark Bar.
7. On the  Develop menu, choose User Agent, then Safari iOS x.xx – iPad
8. Hide the the Menu Bar again.
9. Here’s the painful part: Go 1024 x 768 on your screen resolution. I know, it’s horrible. Do it, and maximize Safari.

After Safari is on your machine and you’re running in 1024 x 768, you can begin developing,

Basic Guidance

Remember, no Silverlight. The fancy PowerPivot Gallery visualizations like Theater, Carousel, etc. don’t work on the iPad since iOS doesn’t support Silverlight and Flash. Make sure you set the default view of the gallery to “All Documents”.  If you don’t want to go this route, you can create a Web Part page for each workbook and add an Excel Web Access Web Part to display the workbook. 

Assume that users don’t know iPad navigation tricks. That’s right, believe it or not some of your users may be dumb. They have an iPad, but they don’t know how to use it. I know, I know – could never happen. If it does happen, they’ll get hopelessly lost trying to navigate your report, and just give up. So throw ‘em a bone with some quick movement tips.

Consider adding a new worksheet to the front of your workbook which explains the two-finger scroll gesture. Some sample text:

To navigate the reports in this workbook:

• Use the worksheet tabs at the bottom of the browser to move from worksheet to worksheet
• To view different parts of a worksheet, press and hold two fingers to the iPad screen, then scroll while keeping your fingers on the screen

Move slicers to the bottom of each worksheet. Horizontal slicers use up valuable screen space and will nearly always force your users to scroll. First, shrink these suckers up so they use less vertical space. Next step, move them to the bottom of the sheet unless it’s absolutely necessary you keep them at the top.

Remove logos and artwork in the report header. Artwork and/or a logo in the header of a report adds a dash of class. However, on the iPad, it makes it less likely you’ll see all the information you need on one screen without a scroll.

Expand row-level groupings in your PivotTables. I like to collapse all the grouped rows in my PivotTables to save on vertical space. While this generally is a good design decision, it doesn’t take into account a mantra that is critical for you to understand in iPad-land. Repeat after me:

The iPad soft keyboard is evil. Do everything you can to prevent it from getting in the way of your user

Each time your user has to interact with a + or – symbol to expand or collapse a grouping of rows, iPad’s keyboard will appear. This gets really old after about the first 10 seconds. Before you save the workbook to SharePoint, expand all your row groupings so users only have to scroll the workbook to view the information it contains rather than do a scroll + “tap” (on the row).

You might also want to add another “navigation note” about row groups to your “how to navigate” cheat sheet:

• To expand or collapse a group of rows or columns, tap and briefly hold the + or – symbol. The web report will re-paint 2-3 seconds later

Consider replacing slicers altogether. When one interacts with a slicer in any way, the blasted soft keyboard will make an appearance. In addition, there are no iPad gestures you can use to easily select multiple slicer items like one can by CTRL and SHIFT clicking in Windows. The “old-fashioned” Report Filters aren’t as pretty or smart as Slicers, but one can select multiple items inside them and they only pop the soft keyboard once when one opens the filter dialog page.  In some cases, you’re just going to need to bite the bullet and use them instead of slicers.

Other Ideas

Turn off the Excel Toolbar: The Excel toolbar becomes something of an appendix when your PowerPivot workbook is viewed on the iPad. Why not remove it and get a few extra rows in your worksheet displayed? You can do so by displaying the workbook inside an Excel Web Access web part, and then turning off the toolbar:

Modify your Master Page: Even a basic SharePoint site generally uses (wastes?) vertical space with header logos, breadcrumb trails, and alike.  If you have the CSS chops, why not get into the master page and modify some of this to free up more room?

Unfortunately, it doesn’t look like it was fixed. A colleague at Microsoft was kind enough to test this, and he still sees the same problem. So, word to the wise: don’t install IE9 on the SharePoint app servers which will be doing snapshot generation.

The Association Rules algorithm focuses on itemsets, which you can think of as a bundle of “things” (attributes) which frequently go together. For example, a popular itemset might be {Peanut Butter, Jelly, Bread}. Itemsets don’t have to be related to shopping, however. A perfectly valid itemset might be {Red Sox Fan, Cretin, Convict}: We’re just creating groups of attributes which go together.

Given many itemsets, the Association Rules algorithm discovers most frequently occurring ones. Then, it creates association rules using those frequently occurring itemsets.  Here’s what a rule looks like:

{Peanut Butter = Existing , Jelly = Existing => Bread = Existing}

If we see Peanut Butter and Jelly in someone’s shopping cart, they are likely to buy Bread, as well.

The concepts of support, probability, and importance are used to tell us how many times we see an item or itemset, the confidence we have in the rule, and the “lift” we might expect for a rule.

If you’d like to understand more about these concepts, read this thread by Jamie Maclennan.

Here’s a basic query to return information from our data mining model:

SELECT * FROM OpenQuery (SSAS,’SELECT NODE_CAPTION as [Rule], NODE_SUPPORT AS [Support], MSOLAP_NODE_SCORE AS [Importance],

NODE_PROBABILITY AS [Probability] from

[Association].content WHERE NODE_TYPE=8′)

Executing this puppy will return all our rules and their support,  probability & importance values. We’re looking for rules with relatively high importance (> 0) which indicate that the existence of peanut butter and jelly will impact bread buying positively:

If you open the image above, you’ll see that each rule looks pretty much like a “formula”. To make these formulae useful, we’re going to need to parse them into individual fields that be used as labels on an axis. To do that, we write lots of kludgy string parsing code:

SELECT

    [Rule],

    — Cast NTEXT to nvarchar then determine number of items in itemset by keying on number of = symbols before the -> characters

    — Divide by 2 since DATALENGTH against NVARCHAR will be twice the string length

    (DATALENGTH(LEFT(CAST([rule] as nvarchar(300)),CHARINDEX(‘->’,cast([RULE] as nvarchar(300)))))- DATALENGTH (REPLACE(LEFT(CAST([rule] as NVARCHAR(300)),CHARINDEX(‘->’,[RULE])),’=',”)))/2   as [ItemSet Size],

    [Support],

    [Importance],

    [Probability],

    — Grab first item in itemset

    LEFT(CAST([rule] as nvarchar(300)), CHARINDEX(‘=’, CAST([rule] as nvarchar(300))) -1) as [Basket Item 1],

    — If two items exist in itemset, grab second using kludgy string parsing

    CASE (DATALENGTH(LEFT(CAST([rule] as nvarchar(300)),CHARINDEX(‘->’,cast([RULE] as nvarchar(300)))))- DATALENGTH (REPLACE(LEFT(CAST([rule] as NVARCHAR(300)),CHARINDEX(‘->’,[RULE])),’=',”)))/2

        WHEN 2 Then SUBSTRING(CAST([rule] as NVARCHAR(300)),CHARINDEX(‘,’,CAST([rule] as nvarchar(300)))+2, ((CHARINDEX(‘->’,cast([RULE] as nvarchar(300)))-12) – (CHARINDEX(‘,’,CAST([rule] as nvarchar(300)))+2) ))

    END as [Basket Item 2],

    — Recommended Item

    REPLACE(RIGHT(CAST([rule] as NVARCHAR(300)), (DATALENGTH(CAST([rule] as NVARCHAR(300)))/2) – (CHARINDEX(‘->’,cast([RULE] as nvarchar(300)))+2)),’ = Existing’,”) as [Recommended Item]

FROM

OpenQUery (SSAS,’SELECT NODE_CAPTION as [Rule], NODE_SUPPORT AS [Support], MSOLAP_NODE_SCORE AS [Importance],

NODE_PROBABILITY AS [Probability] from

[Association].content WHERE NODE_TYPE=8′)

Writing the SQL above was by far the ugliest bit. On to the results:

The resultset above adds “Basket Item 1”, “Basket Item 2”, and “Recommended Item” fields by parsing the the rule column.

Now for the fun stuff. First, let’s look at our rules by Importance and Probability:

Next, lets look at the actual products which are being recommended:

Support is being used you determine our bubble size, so we can see below that Mountain Bottle cages are recommended by 5 different rules:

By highlighting a specific item (product), we can see that while we generated lots of rules which recommend it, those rules aren’t very important – I’d probably ignore these.

Finally, let’s see which items  tend to go together in the same basket. Here we can see that the Mountain Bottle Cage gets grouped with a lot of other items.

I’ve been too busy following my bliss to mess around with it until recently,  but I took the plunge late last week. The authors have implemented the design patterns for a WP7 app nicely, and you’ll see all the things you’d expect, like animated page transitions, etc.

The Basics

The application displays one visualization per page, allowing the user to move between pages of interest. For a free application, I was quite impressed by the number and richness of the visualizations it supports. A look at the Nitro’s  (fairly basic) documentation reveals there are 20 different types you can play with:

• Line Chart
• Bar Chart
• Column Chart
• 1 Line + 1 Column Chart
• Stacked Column
• 100% Stacked Column Chart
• Area Chart
• Stacked Area Chart
• 100% Stacked Area Chart
• Pie Chart
• Funnel Chart
• Bubble Chart
• Pyramid Chart
• Smoothed Line Chart
• Doughnut Chart
• Radar Chart
• Scorecard
• Map with gradient shading.
• Map with KPI shading (green/yellow/red).
• Bullet Chart Scorecard

Some of the animations I noticed in the demo videos looked awfully familiar, and it quickly became apparent that BlueGranite decided to leverage Visifire for most of their charting needs. Hooray! I love Visifire’s controls, and the fact that they have a web-based chart designer is a big bonus. You’ll see why later.

Configuring the application to point at one of BlueGranite’s demo dashboards is very straight-forward, and between the two built-in samples and AdventureWorks, you’ll have more than enough to play with.

Creating your own Dashboard

One feeds the dashboard data using XML. You can leverage a static XML file on a web server to do so, or render a Microsoft SQL Reporting Services report as XML to provide a dynamic data feed. Well, you know what I’m going to do. Let’s roll up our sleeves!

BlueGranite provides a walk-through on how the AdventureWorks sample SSRS feed was created, and I’m glad they did. I advise you to take your time with all the files that make up the sample and get really familiar with how BlueGranite did what they did. I found that creating one of these from scratch is not for the faint-of-heart. You’re going to be writing plenty of TSQL and possibly hand-crafting some MDX before you’re done. You’ll also want their XML reference documentation.

I decided to create a dashboard using sample online analytics data I have for a fictional ad agency named WWReach Advertising. I’ve used the same data in other examples on this blog. All of that lovely data lives in a cube.

Queries. Lots of them.

The first thing you’ll do is write a “Control Panel” query which pretty much tells Nitro:

• How many pages are going be displayed in your dashboard
• Which type of visualization will be displayed on each page
• How each visualization should display and format data

This SSRS dataset which contains the query above must be named DataLayoutSet, and it will contain one SELECT statement for each page in your dashboard. If you have 5 pages, you’ll write 5 SELECT statements and use UNION to string them altogether inside the dataset.

I recommend you start with one and only one SELECT in DataLayoutSet. Don’t put 2-3 in at the same time like I did – too many chances for error.

After you’ve created DataLayoutSet and it contains details for at least one dashboard page, you’re going to create more queries which actually grab your data. I assumed this part would be really simple, but it wasn’t.

BlueGranite Nitro requires that the queries which feed a dashboard page be shaped in a particular manner. Based on the type of visualization you want to use, it’s likely you’ll end up having to jump out of the graphical query designer and start hand-coding. The graphical designer isn’t going to be able do things like create MDX Members that you’ll need if you happen to use SSAS as a data source like I did.

Now that the “data query” is written, you’ll need to modify the field names in its containing dataset to match the attributes you specified in DataLayoutSet. You’ll name these suckers x1, y1, y2, z1, etc. Why?

Visifire. Visifire does it’s data plotting based on values supplied to  ‘”X Value”, “Y Value” and “Z Value” attributes. If I want to plot Sales and Profit Margin by Month, x1 will need to display Month values, y1 will show Sales, and y2 spits out Profit Margin. This is where the Visifire chart designer I mentioned earlier comes in really handy. After you run a query, just take the values it displays and plug ‘em into the chart designer. You can very quickly get a feel for what your page is going to look like inside BlueGranite Nitro even if your SSRS report isn’t completely working yet.

OK, enough. The gents at BlueGranite wrote a nice walkthrough, so use it. Here’s mine:

I’ve added my dashboard using the options screens. Want to use my sample? Here is the RDL file for you to take a look at. You might just be interested in a static XML file, too. 

You can see that my scorecard doesn’t have tons of room for multiple columns. You’ll need to be choosy about what you put here.

The area chart above shows Click Through Rate for 4 clients (Big Air Domestic, Big Air International, EuroRail, etc.) by Vendor (Accuweather, AdBrite, etc..).  Again this is fictional data. Make believe. Fake.

One problem in the current version of Nitro appears to be that it doesn’t handle formatting small numbers very well. Click through rates often range between a tenth to half of a percent, and we’re having a hard time displaying that above.  Attempts to format these values with ‘P’ (Percent) didn’t do anything.

Here’s the same page in landscape mode:

The final page I dropped into my dashboard was a bubble chart. I love my bubble charts!

In the visualization above we’re plotting clicks and cost per click for my four clients. In a perfect world, I’d also want to vary the size of the bubbles based on the number of impression each client was getting. Visifire does support this ability, but it hasn’t been implemented in Nitro yet. I understand it’s coming soon, and I can’t wait!

Roundup

This is a great first effort from BlueGranite. They deserve lots of kudos for contributing to the MSBI ecosystem with this super little tool. And now that I’ve delivered a few compliment and hopefully have them well and truly buttered up…I beg for more features.

What would I like to see?

• oData as a potential data source. I’m playing around with SQL Azure quite a bit and being able to consume oData would be quite handy
• Ability to control the size of bubbles in a bubble chart
• Ability to create custom captions for my bubbles, bars, etc.
• More cowbell!

I’ve thrown “my” sample out on the internets, so if you want to connect to it and play, here’s where you can find it:

URL: http://wwreach.msftlabs.com:8082/reportserver_PowerPivot?/MobileStuff&rs:Command=Render&rs:Format=XML (…and I apologize…this URL is MURDER to type in on a soft keyboard)

Credentials: UserName=corp\runreport Password=pass@word1

Again, many thanks to the BlueGranite for helping to build Microsoft’s MobileBI story!

One will generally use the clustering algorithm to segment “things” (customers into “customer clusters”, for example) based on similarities between attributes that describe the “things” in question.

The Adventure Works sample cube has a ready-made data mining structure which contains a customer-focused clustering model. It’s called Customer Clusters and takes the following inputs:

• Commute Distance
• Education
• Gender
• Home Owner
• Marital Status
• Number of Cars Owned
• Number of Children at Home
• Occupation
• Total Children
• Yearly Income

Rather than recreate the wheel, I’ll use the sample.

The first thing we’ll probably want to understand is how different combinations of attributes and attribute values combine to create distinct clusters of customers – otherwise known as a “cluster profile”. SQL Server Analysis Services has a quick but unsupported system stored procedure named GetClusterProfiles() which will give us this information, and since it’s the weekend, I don’t care if it’s unsupported. I’m going to execute this sproc against my data mining model:

Click and view the image above as there’s a lot of good information to be found. The columns 001-010 represent 10 distinct clusters of customers that were discovered based on the data the model was trained on. Attribute Name and Attribute Value show the inputs and potential input values that were found in the data. For each attribute name/value pair, you’ll see a support and probability value. Support represents the number of cases (rows) we found that fit the criteria in question. Probability tells us the probability a particular attribute name/value pair will show up in the customer cluster in question

For example in Customer Cluster 001, there is a ~ 42% chance the customer lives less than a mile away from work, about a 1% chance the person lives 10+ miles away from work, and a ~25% chance they live within 1-2 miles, etc.

As an aside, it’s normal to actual rename clusters from 001, 002, 003 to something more meaningful like “Soccer Moms” , “Yuppies” or “Hipsters” based on our read of the primary drivers of each cluster. I haven’t bothered, so we’ll refer to our clusters as 001-010.

As I mentioned, I initially wanted to use GetClusterProfile()’s output as my data source, but found that I couldn’t use OPENQUERY and the Microsoft MSOLAP provider to execute it. I’m guessing that the fact we return an unknown number of columns (based on the number of clusters we find) could be causing the problem.

Instead, I had to use multiple DMX statements “UNIONed” together. Each DMX SELECT returns cluster attribute name/value pairs for a single attribute. Here’s one for Commute Distance:

There’s something important here to note, by the way. Notice the WHERE clause where I filter out any attributes with less support than 500 cases. Choosing 500 was an arbitrary decision on my part, but it impacted my results in a big way. In the output above, there are now only 3 Commute Distance nodes returned for customer cluster 001, while there were 5 when I executed GetClusterProfile() in the first screenshot. Is this good or bad? Hard to say – there were only 35 cases supporting the “1% probability of living 10+ miles away” node in cluster 001. Is 35 statistically significant enough to include?

(By the way, after the whole discussion around SUPPORT vis-à-vis “Customer Commute” I managed to completely FORGET to add it to the final query above! Since this is just an example I decided not to go back and correct my oversight, however – don’t be surprised when you don’t see the Commute Distance attribute in any of the following vizes.)

After dropping our final SELECT statement into a Tableau “Custom SQL” data connection, we’re ready to build a Viz:

• The Node_Name (The name of the cluster: 001, 002, etc.) pill lands on Columns
• Attribte_Name and the Probability measure get dropped in Rows
• Attrbute_Value lives in the Color shelf

You can now easily see how each attribute value drives the composition of different customer clusters (001-010, on columns).

I found there were mixed text and number values in the Attribute_Value column, and this was painful. For example, I had to deal with values like “Married” and “Single” (Marital Status) and .056365 (Number of Cars Owned in cluster 010) in the same field. I used Groups to simplify things a bit, but I’d probably massage the data model / query a bit more if I were doing this “for real” rather than “for fun”.

After we understand how customer clusters are composed, we can really have some fun.

The Adventure Works sample SSAS database contains a second cube called “Mined Customer”. Mined Customers combines two data mining models (Customer Clusters and an Association Rules model for market basket analysis) with lots of other metrics like sales, profit. The mash up of our “default” metrics and dimensions with the “mining insights” gives us some exciting possibilities for visualization.

In the viz below, we’re able to look at average sales and profit margin by our ten customer clusters. One can see that Cluster 5 drives the highest profit margin with a relatively large number of customers:

And the fun continues! By dropping one of the “demographic inputs” on the Level of Detail shelf, we can begin exploring within individual clusters. I’m going to drill in by Occupation.

It now becomes apparent that skilled manual workers in Cluster 6 are more profitable than Cluster 5 as a whole. By putting together my Quick Filters and Demographic fields on the Level of Detail shelf I also have the ability to completely explore the sales and profitability of my customer clusters.

More Cowbell

And I have an admission to make: I’m obsessed. Since Gartner released its 2011 Magic Quadrant for Business Intelligence Platforms report a few weeks ago, I’ve had MQ on the brain. I’ve been thinking about MQs non-stop – it’s my cowbell. In fact, I got a fever, and the only prescription…is more cowbell!!”

So, I decided I’d take the previous viz and turn it into a MQ. I essentially zoomed in a tad, and dropped in four annotation areas with no labels and played around with background shading. Voila, more cowbell!

Next time, Association Rules – otherwise known as market basket analysis

One thing that has occurred to me as a potential bump in the road towards wider adoption  is visualization. Microsoft offers some sample (and admittedly long-in-the-tooth) data mining visualizations but perhaps users still find it too hard to get their newly won insights out of SQL and/or Excel and present same to rest of the world? Maybe users want more control over how the visualizations look and feel and how one interacts with the results?

To that end, I thought it would be fun to work with a product from Microsoft gold certified partner, Tableau Software. The folks at Tableau make a pretty awesome piece of BI goodness named Tableau Desktop that has really, really nice visualization capabilities. Tableau Desktop is a bit like my old favorite ProClarity, except that it can consume data from SSAS to SQL to Teradata to Netezza to PowerPivot and back to text files. Did I mention the tool has awesome visualization capabilities?

What I’m going to do over the next few weeks or as time allows is to drop Tableau on top of SQL Server Analysis Services data mining output. I’ll try to come up with meaningful examples of how one might visualize the output of all the major data mining algorithms SSAS supports. Sounds fair?

Time Series

I’ll start with the Time Series algorithm, which we’ll use for predicting values based on numbers tied to slices of time in the past. One essentially feeds a Time Series model a bunch of numbers which are each associated with a timestamp, and SSAS can come back to you with predicted values for other time periods.

In PerformacePoint 2007, we took advantage of this functionality directly and had a really cool report type which one could nest in a dashboard. Unfortunately, the report type leveraged the deprecated Office Web Components, so we don’t have it PPS 2010. In honor of my lost friend, we start with the Time Series:

I’m going to assume you know something about Microsoft data mining. If you don’t, I advise you to spend some time with the fine tutorial and books online. As much as I can, I’ll actually base my work off these walkthroughs and examples in the sample AdventureWorks cube.

So, we’ll start off in BIDS with a simple DMX query which predicts the next 6 periods (months) of sales based on history:

I’m re-using the “Forecast” model that is already waiting for us inside the SQL Serve Analysis Services sample cube AdventureWorks DW 2008R2. The query will return the model of item we’re selling, the region it’s being sold in and 6 months worth of predictions.

The “as-is” query has a problem in that the results come nested as distinct datasets.  This is pretty hard to use :

We’ll fix this annoyance by leaning on the DMX FLATTEN statement. One can flip from ‘Designer” to “SQL” view and modify the query itself:

SELECT FLATTENED

  [Forecasting].[Model Region],

  (PredictTimeSeries([Forecasting].[Amount],6)) as [Amount],

   True as [Predicted]

From

  [Forecasting]

Note how I’ve also added a hard-coded field – “Predicted”; I’ve set the value to “True” to indicate that the values in question are predictions, not historical values.

The look that we’re ultimately going for in our visualization is a single, unbroken line or bar chart which gradually transitions from plotting historical values to predicted values. The bit above gets us our “in the future” numbers.

Getting the historical numbers is pretty easy because there happens to be a SQL Server view which provides them to the data mining algorithm for training purposes. The view is named vTimeSeries, and I’ll just re-purpose it:

SELECT ModelRegion,

   TimeIndex,

   Amount,

   0 as [Predicted] –-‘False’: Historical values

FROM vTimeSeries

How do we merge these two distinct sets of data? UNION:

SELECT * FROM OpenQuery (SSAS,’SELECT FLATTENED

  [Forecasting].[Model Region],

  (PredictTimeSeries([Forecasting].[Amount],6)) as [Amount],

   True as [Predicted]

From

  [Forecasting]‘)

UNION ALL

SELECT

   ModelRegion,

   TimeIndex,

   Amount,

   0 as [Predicted]

FROM vTimeSeries

Note that we use OpenQuery to make this all work – we therefore have a Linked SQL Server Analysis Server plugged into our SQL Server. I’ve named it SSAS, and here’s the property dialog:

Once the linked server is there, we can test our query:

Now that the query is done, the real fun can begin. In a new Tableau workbook, we’ll need to create a new Data Connection pointing at SQL Server and using the Custom SQL option. Paste the query above into the Custom SQL dialog, and you’re pretty much ready roll.

• Drop your Time dimension on Columns
• Drop ModelRegion and Amount on Rows
• Place the “new” Predicted dimension on the Color shelf.

If you click the image above, you’ll see that the Predicted field drives the historical/predicted value transition via a color change.

I also added a few Calculated Fields to break out “Region” and “Model” into two distinct dimensions and added each as a filter. For kicks, I changed my shapes from bars to prediction-or-not influenced squares and circles:

That’s it!

While testing that I could return results from DMX via OpenQuery, I ran a basic DMX statement to return results from a time series model:

SELECT * FROM OpenQuery (SSAS,'SELECT FLATTENED
  [Forecasting].[Model Region],
  (PredictTimeSeries([Forecasting].[Amount],3)) as [Amount],
   True as [Predicted]
FFrom
  [Forecasting]')

Doing so returned this error message:

   OLE DB provider "MSOLAP" for linked server "SSAS" returned message "Error (Data mining): The '\\?\C:\Program Files\Microsoft SQL Server\MSAS10_50.MSSQLSERVER\OLAP\Data\Adventure Works DW 2008R2.0.db\Forecasting.0.dms\Forecasting.0.dmm\0.Forecasting.cnt.bin' persisted file cannot be found.".
Msg 7321, Level 16, State 2, Line 1
An error occurred while preparing the query "SELECT FLATTENED
  [Forecasting].[Model Region],
  (PredictTimeSeries([Forecasting].[Amount],3)) as [Amount],
   True as [Predicted]
From
  [Forecasting]" for execution against OLE DB provider "MSOLAP" for linked server "SSAS".

Problem? Stupid user trick – I hadn’t processed the model.  I was surprised to find that this error message couldn’t be found via Bing/Google, however. So here’s a little entry to save someone else some time.

Cheers.

When playing inside a PerformancePoint Services 2010 scorecard, you may encounter an odd behavior from time to time: You right-click an area of the scorecard in order to expose the context-menu, and the menu doesn’t appear under your mouse. The menu does indeed get launched, but it is displayed in the wrong place – generally in the upper-left hand corner of your browser:

Neat, huh?

The issue seems to do with browser compatibility. 99% of the time, if you turn the Internet Explorer Compatibility View feature own, your problem will be solved.