While watching the keynote from the recent BI Conference, I saw a demo of the Pivotviewer Extensions for Reporting Services. This is an interesting tool that will help automate creating Pivot collections. Unfortunately, even as an Microsoft FTE I can’t get my hands on a preview copy until after the end of the month…and I have a project due before then.

So, what the hell, let’s re-create the wheel! You can download some samples here. The samples aren’t a complete solution, but demonstrate the uglier/trickier bits of the process.  The code is ugly and not commented…you get what you pay for.

Goal:

Create a collection which surfaces online advertising metrics (impressions, clicks, click through rates, etc.) for 10 ad agency clients. Each client has multiple campaigns and placements with a number of different packages, ads, and creative items which are served by multiple vendor websites.  

Create a Pivot “card” for each combination of Client-Campaign-Placement-Ad-Site . Each “card” should also display interesting metrics on a day-by-day basis.

All data is to be sourced from an SSAS cube.

Approach:

Short and sweet: Create an SSRS report and code to repeatedly export a JPEG in order to generate the images used in the Pivot collection. Once complete, use the Pivot command-line tools to create the cxml document. Simple, right? Here’s how I did it:

Pull information from SSAS

First, I “wrote some MDX” to pull dimension attributes (like client, campaign, etc.) and metrics (impressions, clicks, click through rate)  from the cube.  I had to add additional fields to this query to return the actual UNIQUENAME of the dimension members I was returning. Why? You’ll see in the next section…

WITH          
MEMBER [Measures].[ClientNameUNIQUE] AS [Clients].[Client Name].CURRENTMEMBER.UNIQUENAME          
MEMBER [Measures].[CampaignNameUNIQUE] AS [Campaigns].[Campaign Name].CURRENTMEMBER.UNIQUENAME
MEMBER [Measures].[AdNameUNIQUE] AS [Ads].[Ad Name].CURRENTMEMBER.UNIQUENAME
MEMBER [Measures].[SiteNameUNIQUE] AS [Ads].[DFA Site Name].CURRENTMEMBER.UNIQUENAME          
        
SELECT {[Measures].[ClientNameUNIQUE], [Measures].[CampaignNameUNIQUE] , [Measures].[AdNameUNIQUE], [Measures].[SiteNameUNIQUE],
 [Measures].[Impressions],[Measures].[Clicks],[Measures].[CTR],[Measures].[Calculated Cost],[Measures].[Direct Sales], [Measures].[Indirect Sales]} ON COLUMNS,
NONEMPTY(         
{ { { [Date].[Year].[All].CHILDREN } *
{ [Clients].[Client Name].[All].CHILDREN}  *
{ [Campaigns].[Campaign Name].[Campaign Name].ALLMEMBERS} *
{ [Placements].[Target Market].[Target Market].ALLMEMBERS} *
{ [Placements].[Package Name].[Package Name].ALLMEMBERS} *
{ [Ads].[Ad Name].[Ad Name].ALLMEMBERS} *
{ [Ads].[DFA Site Name].[DFA Site Name].ALLMEMBERS} *
{ [Creative].[Technology Type Name].[Technology Type Name].ALLMEMBERS}

 }}

, [Measures].[Impressions]) ON ROWS           
FROM [WWReach]

I wrote a simple SSIS package (PopulatePivotSupport.dtsx) to take the resultset from this query and drop it into SQL for staging purposes.

Create SSRS report

Next, I created a parameterized SSRS report which contains a fictional client logo, as well as multiple SSRS charts which are used to plot daily online metrics for the client/campaign in question.

The report is parameterized so that I can feed in arbitrary values used to filter data in the charts.  My report uses 5  parameters to filter by  client, campaign, ad, (vendor) site, ad serving technology, etc.

Here’s an example :

A sample RDL can be found in the files I mentioned at the top of the article.

Challenge #1

Things normally don’t go 100% smoothly for me when I need to do heavy parameterization on a report which uses SSAS as a data source. I normally have issues with StrToSet-related errors, trying to plug in values which SSRS doesn’t see as valid (due to CONSTRAINED flags in my MDX), etc.  To try and avoid as many of these problems as possible, I did the following for each parameter in my report:

• Deselected Allow multiple values
• Under Available Values, I chose None instead of Query
• Double-checked that I only had one default value per parameter (see bullet #1)
• Removed the CONSTRAINED sections of MDX from my queries

Challenge #2

I need to provide parameter values to SSRS/SSAS in the [dimension].[attribute].&[attributekey] format.  That’s why I created several MEMBERs in my MDX. I used those to return not only a dimension attribute value like “Us Rail” (a client’s name), but the client’s Unique Name: [Clients].[Client Name].&1001. This was the critical step. Life was good once I had all the “friendly” values (like Client Name = “Us Rail”) as well as the “not so friendly ones” that I needed to feed back to SSAS for filtering purposes ([Clients].[Client Name].&1001) in a staging table.

Write code to render (repeatedly) reports to JPEG files

I decided to create another SSIS package named  PrintSSRS.dtsx to do this work. Essentially, the package does the following:

• Grabs rows from the staging table. Each row runs my SSRS report once.  I used an Execute SQL task for this and stuffed the results into an Object Variable
• Uses a For..Each task to iterate through the rows in the Object variable, grabbing values from each row like the Client ID, Campaign Key, Site ID, etc.
• Executes a Script task inside the For…Each loop. The script task populates the SSRS report’s parameter collection and calls .Render() against SSRS to create the JPEG report which is saved to disk.

..I ended up with about 5,500 JPEG “reports” on my hard drive…each one was about 50K. It took about 2 hours in total for this code to generate all the reports I needed.

Generate the Pivot Collection

Since I hadn’t used the (new) Pivot Command-line tools yet , I thought I would give those a try instead of the Excel-Add in. You can download them from http://www.getpivot.com

I still used Excel heavily though:

• First I imported that staging table from SQL into a worksheet.
• I deleted the Excel columns I didn’t care about (the “MEMBER” columns)
• I used the Pivot documentation to create 3 worksheets in Excel which defined my collection
• I ran the Command-line tool to create a deepzoom output and went to dinner.

I’ve dropped the workbook I created into the samples, too.

Other Stuff

I learned the hard way that I should have spent a bit more time with the Pivot developer documents. My first collection actually consisted of over 72K distinct Pivot tiles because I decided to also break down my clients/campaigns/ads/sites/technologies by week. Bad idea – I actually couldn’t get the command-line tools to finish the job of creating the CXML file…they would run out of memory on my 16 GB box before they were through

I next tried breaking things down by month instead of week…I was able to get a working collection of about 15K tiles at that point, but it looked terrible in viewer. So, I threw our the idea of filtering by time altogether, and here is the result (FYI, “City of New York” is purely fictional – I’m a native, so just used them as a make-believe ad agency client):

As I said, I should have RTFM, because it said my collections should try and stick to around 5,000 items.

So there you go. If you’re impatient and don’t want to wait for the “official” PivotViewer Extensions, just roll one yourself. Microsoft’s version is going to be much better than this hack, btw: faster, able to handle more rows, etc.

Over the week-end I was doing some analysis on SQL Server disk usage, and wanted to be able to display current disk usage by database. Up to this point, I’d mainly been doing a SUM over  my measures. Well, that would make no sense in this scenario – I’d end up with a meaningless number just like I would if I took a SUM of an Inventory value.

I found myself dealing with a classic semi-additive measure. These are easy to deal with using SQL Server Analysis Services, but a tiny bit trickier in PowerPivot.

I tried various DAX-related ideas, and some actually kind of worked, but Marius Dumitru, a co-worker at Microsoft, suggested the best solution.

I happen to have a date dimension which we could lean on for this challenge – I just needed to find the last “Disk Use” value that had been recorded. Since all of my readings are associated with a date, slam dunk!

Here’s what I did:

Created a calculated measure which SUMs the measure in my table (I named it “Sum Disk Size”):

     = SUM(‘Fact_DatabaseMetrics’[Disk Size])

Created a second calculated measure which filters the first, looking for the value related to the last associated date in my date dimension (via DAX’s LastNonBlank function) . I didn’t know you could filter a measure so easily!

        = ‘Fact_DatabaseMetrics’[Sum Disk Size](LastNonBlank(‘Dim_Date’[DatePK], ‘Fact_Database’[Sum Disk Size])

The second measure is what I added to my PivotTable.

If you’ve played with PowerPivot at all, it’s pretty obvious how flexible the tool is in terms of creating data models. After you initially create a model, you may want to spend some additional time with it to make sure users can easily leverage what you’ve created in other tools. If you don’t, the PowerPivot model you publish probably won’t see lots of re-use outside of the PowerPivot gallery and Excel because users won’t be able to get at the values they want to.

What am I talking about? Here’s an example:

I pointed PowerPivot at the Adventure Works DW sample database, and brought all its tables and rows into my workbook. I then published the workbook directly into a SharePoint 2010 PowerPivot Gallery. I didn’t bother to add any UI to the sheets in the workbook (no PivotTables, PivotCharts, Slicers).

Next, I launched Report Builder 3.0, created a new Analysis Services data source and typed in the location of the *.xlsx I published. I chose the single “sandbox” in the workbook as the database name. Here’s a screenshot of my new RB data source being created (embedded image truncated, click below to view complete screenshot) :

Then, add a DataSet to this Data Source and launch the Query Designer. Notice anything wrong with this picture?

That’s right — our dimensions are showing up as measure groups AND dimensions! And if you scroll down the dimension list, you’ll see that the the fact tables not only show up in the measure group list, but as dimensions, too!

Finally, crack open one of the measure groups – It contains NONE of the measures that are defined in the PowerPivot model – only  “_Count FactSalesQuota” in the screenshot below.

What the hell is going on? What you’re seeing is a function of how PowerPivot “thinks”. To PowerPivot, everything is a dimension unless proven otherwise, and therefore all of the facts in your measure groups are considered dimension attributes until PowerPivot understands that the attribute in question should really be considered a measure.

So let’s fix all this stuff up so we have a usable model, shall we?

To begin with, we’re going to need to help PowerPivot figure out which of our fields represent measures. In PowerPivot there are implicit and explicit measures. Any field you drag into the Values list of a PivotTable/Chart is regarded as an implicit measure by PowerPivot.

I’ve opened up that empty AWDW workbook again, and added a PivotTable.  Next, I’ll add the SalesAmountQuota measure from the FactSalesQuota table to the Values area of the Field List (embedded image truncated, click below to view complete screenshot).

After publishing the updated workbook to MOSS, let’s look at Report Builder again.

There you go – an implicit measure. By adding a field to the Values list of the Gemini Task pane, we’ve tipped PowerPivot off to the fact that SalesAmountQuota is indeed a fact.

However, if you look at the dimension list, you’re still going to see FactSalesQuota as a dimension, and SalesAmountQuota (along with 4-5 other fields) as a possible dimension attribute. Well, that ain’t right.

How to fix? Make that measure an explict measure. Here’s what to do.

Remove the SalesAmountQuota from the Values List in your PivotTable.

Switch back into the PowerPivot window, find the FactSalesQuota table, and select all the columns. Finally, right-click the selected columns and choose to Hide Columns | From Gemini and PivotTable

Back in the worksheet, click Refresh All on the Data ribbon, and your field list should look something like this (note how our measure group name is “dimmed”  – embedded image truncated, click below to view complete screenshot):

Right-click FactSalesQuota, and select Add New Measure. Create a formula which points to the SalesAmountQuota field you hid just a second ago (embedded image truncated, click below to view complete screenshot):

When you’re done, you should have something that looks like this (embedded image truncated, click below to view complete screenshot):

That’s just about it. Publish your workbook again, and fire up Report Builder. Note that the explicit measure you added now shows as part of the measure group FactSalesQuota and that FactSalesQuota doesn’t appear at all under dimensions.

And some notes:

In order to get the data model to refresh in Report Builder as I made changes in Excel and re-published, I found I had to sometimes delete the published workbook and/or IISReset. Using the Reconnect/Refresh button in Report Builder’s SSAS query designer didn’t refresh my schema for some reason.

Many thanks to Oliver Matrat, a senior PM on the Analysis Services team for helping me out when I was confused about how to make this stuff work – most of what I’ve related here comes directly from him.

I had a bear of a time getting SQL Server 2008 R2 November CTP’s Integrated PowerPivot feature installed.  It looks like several other people in the Twitter/Blogosphere are running into the same issue, but for potentially different reasons.

For me, ProClarity Analytics Server (PAS) 6.3 was the culprit: If it was already installed on my machine, I could NOT get Integrated PowerPivot installed. As a result, I was forced to install Gemini first, then PAS. In order for everything to play nicely, I then needed to manually move the PAS install from port 80 into a new website which listened on a different port. How did I do it? Here’s how:

WARNING: Before you install PAS, I suggest you read the couple of sentences & 3 bullets towards the end of this entry – you are about to inadvertently (but temporarily)  BREAK MOSS by changing all of MOSS’s x64 app pools to x86/x64. I suggest you record a list of all your applications pools as they exist right now and note whether they Enable 32-bit Applications or not so you can make sure they have the correct values when you’re all done.

• I set the identity of IIS’s DefaultAppPool to my domain admin account to make life simple. You might not want to.
• Install PAS as you normally would, including all the extra goodies that come with PAS like ProClarity Web Pro, etc.
• Install PAS 6.3 Service Pack 3 http://www.microsoft.com/downloads/details.aspx?FamilyID=E6B784A8-E7C2-4E27-9D98-41E2F2FD7467&displaylang=en
• Install WebDAV. Here’s a link which gets more specific (see section “Installing and configuring WebDAV”): http://office.microsoft.com/download/afile.aspx?AssetID=AM103875331033
• Open IIS Manager via Start  | Run | inetmgr
• Create a new website named ProClarity. Bind it to port 81. Along with it, you’ll be creating a new AppPool named ProClarity, as well.
• Modify the AppPool named “ProClarity“, set it’s Framework Version to v2.0.50727, Pipeline Mode = Integrated. Set the Identity of this pool to your domain admin account or something more secure.
• Copy the PAS and PASUploads applications plus the ChartFXInt62 VDIR currently under Default Web Site to the ProClarity website
  • Under Default Web Site, right-click the PAS application, choose Manage Application, then Advanced Settings. Copy the Physical Path string to your clipboard
  • Right-click the ProClarity web site, choose Add Application, and create a new application with the same name/alias (PAS). Paste in the Physical Path string.
  • Repeat for PASUploads
  • Repeat for the ChartFXInt62 Virtual Directory, but create a VDIR under the ProClarity web site instead of an application
• Browse to C:\Windows\System32\inetsrv\config and back up ApplicationHost.config somewhere safe
• Open ApplicationHost.config and scroll all the way to the bottom of the file until you hit the closing </configuration> element.
• Paste the following stuff right before the closing </confiiguration> tag. I’m assuming you named your website ProClarity, btw:

     <location path=”ProClarity”>
             <system.webServer>
                <isapiFilters>
                     <clear />
                     <filter name=”PHttpFilter” path=”C:\Program Files (x86)\Common Files\ProClarity\Server\PHTTPFilter.dll” enabled=”true” />
                 </isapiFilters>
             </system.webServer>
         </location>
         <location path=”ProClarity/PAS”>
                  <system.webServer>
                 <directoryBrowse enabled=”false” showFlags=”Date, Time, Size, Extension” />
                 <handlers accessPolicy=”Read, Script” />
                 <security>
                     <authentication>
                         <windowsAuthentication enabled=”true” />
                         <anonymousAuthentication enabled=”false” />
                         <digestAuthentication enabled=”false” />
                         <basicAuthentication enabled=”true” />
                     </authentication>
                 </security>
                 <defaultDocument enabled=”true”>
                     <files>
                         <clear />
                         <add value=”ProClarity.asp” />
                     </files>
                 </defaultDocument>
                 <asp enableParentPaths=”false” bufferingOn=”true”>
                     <session allowSessionState=”true” />
                     <limits scriptTimeout=”00:03:00″ />
                 </asp>
             </system.webServer>
         </location>
         <location path=”ProClarity/PASUploads”>
             <system.webServer>
                 <directoryBrowse enabled=”false” showFlags=”Date, Time, Size, Extension” />
                 <handlers accessPolicy=”Read, Write, Script” />
                 <security>
                     <authentication>
                         <windowsAuthentication enabled=”true” />
                         <anonymousAuthentication enabled=”false” />
                         <digestAuthentication enabled=”false” />
                         <basicAuthentication enabled=”true” />
                      </authentication>
                 </security>
                 <defaultDocument enabled=”false”>
                     <files>
                         <clear />
                         <add value=”ProClarity.asp” />
                     </files>
                 </defaultDocument>
                 <asp bufferingOn=”true”>
                     <session allowSessionState=”true” />
                     <limits scriptTimeout=”00:03:00″ />
                 </asp>
             </system.webServer>
         </location>
         <location path=”ProClarity/ChartFXInt62″>
             <system.webServer>
                 <directoryBrowse enabled=”false” showFlags=”Date, Time, Size, Extension” />
                 <handlers accessPolicy=”Read, Script” />
                 <security>
                     <authentication>
                         <windowsAuthentication enabled=”true” />
                         <anonymousAuthentication enabled=”false” />
                         <digestAuthentication enabled=”false” />
                         <basicAuthentication enabled=”true” />
                     </authentication>
                 </security>
                 <defaultDocument enabled=”true” />
                 <asp>
                     <limits scriptTimeout=”00:03:00″ />
                 </asp>
             </system.webServer>
         </location>

        What is this? It’s the basic configration of your Default Web Site with PAS installed (only PAS, nothing else) . I just snipped it out of my config file, and renamed “Default Web Site” to “ProClarity” in the appropriate places).

• Save, and do an IISReset
• Browse to http://machinename:81/PAS, and ProClarity should be happy.
• Assuming all is well, remove the PAS, PASUploads and ChartFXInt62 applications/VDIRS from under Default Web Site.
• Next, select the Default Web Site and click ISAPI Filters in the Features View pane.
• Delete PhttpFilter, which is most likely the only thing there.

If you’re trying to install the Integrated PowerPivot engine for SharePoint, then I know you’re on a 64-bit machine, which leaves you with another problem. Since you just put PAS (32-bit) on your box, IIS has switched all of your Application Pools  around so that they allow 32-bit apps. Your x64 MOSS install won’t like this at all. It is broken. SO, fix it:

• In Internet Information Manager, click Application Pools. You’ll see a bunch of stuff from MOSS including multiple app pools with a GUID for a name, Classic .NET AppPool, DefaultAppPoolk, Sharepoint – 80, SharePoint Central Administration v4, SharePoint Web Services Root.
• Right-click each one, choose Advanced Features, and then set the Enable 32-bit Applications property back to False. (You must leave the ProClarity app pool set to True).
• IISRESET

 That’s it, you should be good. BTW, if you chose not to set the identity of the ProClarity and/or DefaultAppPool to a set of domain admin \ admin credentials, you still might have a bit of work to do to make sure that ProClarity can read everything it needs to. Read this link for more information http://blogs.technet.com/proclarity/attachment/3171346.ashx. You’ll be interested in steps 5 and 6 of section 1.

On a fairly regular basis, customers I work with wonder out loud (read: gripe) why SSAS 2005 and 2008 don’t allow them to easily drill-down to the relational tables that back a cube using an action. SSAS 2000 did this nicely, but SQL Analysis Services 2005 and 2008 want to return rows from the cube vs. directly from the tables.

There is a sort-of-but-not-too-painful workaround available which encompasses using ROLAP partitions which point to the drill targets in question, but it’s still kind of inconvenience to deal with because you may end up setting up quite a few ROLAP partitions to support everything you need to do.

Brian Knight has come up with a really nice, clean solution which leverages a helper assembly to do the same thing. I like this approach much more because once you’ve installed the assembly, creating drill-through actions is quite easy. Thanks, Brian!

http://blogs.pragmaticworks.com/brian_knight/2009/09/creating-a-ssas-rowset-action.html

http://blogs.pragmaticworks.com/brian_knight/2009/09/calling-an-external-query-from-mdx.html

I’m giggling like a little girl because today (well, last night) CTP2 (also known as the “August CTP”) hits the streets.

I jumped out to MSDN and TechNet and see subscribers will be able to download it today – I’m actually looking at the bits out on TechNet right now.  A general release will be available two very long days from now, on Wednesday the 12th.  You can go for here for more information. The download link is here.

So what’s in CTP2? That’s sort of a good news / bad news proposition, actually.  As you know, R2 is hitting on the self-service BI theme with all cylinders. The main focus of this effort will be Project Gemini, but don’t forget about SQL Reporting Services which has much new coolness, too.

Project Gemini

I’m sure you all know about Project Gemini, and if you don’t, you should read about it here. Gemini is part of the next wave of innovation represented in the R2 / MOSS 2010 and Office 2010 releases. It is generally discussed in relation to the SQL 2008 R2 release. However, the CTP2 release of SQL doesn’t actually include the Gemini add-in for Excel 2010 – and most folks don’t have access to Excel 2010 yet, anyway.

The next community tech preview, CTP3 will contain Gemini bits. And, no, I don’t know when it will be released. I’m also unclear if this will be a public CTP or if you have to be invited.  In any case, if you want to sign up for notifications about CTP3, go here:   https://profile.microsoft.com/RegSysProfileCenter/wizard.aspx?wizid=27d33a54-f9f5-4357-b652-6f03ba8a4d23&lcid=1033.

For those of you lucky enough to be part of the Office 2010 Tech Preview, you can get your hands on a CTP2 synced version of the add-in right now. More information. You’ll see links at the very bottom of  the page that allow you to register and then download.

SQL Reporting Services

In terms of SSRS, you’re going to be able to play with Report Builder 3.0. It includes the new mapping functionality, which is still a work in progress in CTP2. It’s quite cool, however.

For all you recovering Microsoft Access junkies: Remember the love-hate relationship you had with DLoopkup? The function everyone maligns and then uses anyway? Well, SSRS has got it! Ours will be called Lookup(), and I won’t spoil the fun…check out Lookupset() and MultiLookup(), too.

Some really compelling functionality isn’t in the CTP yet, and I’m not sure whether the SSRS team has talked about it yet, so I’m not going to name/whine about its absence and inadvertently let the cat out of the bag.  Additional good stuff is coming, though!

Books Online

Don’t forget to RTFM! Download it here.

Carl Rabeler of the Microsoft SQLCat team has just released a really nice set of tools targeted at helping you analyze SSAS performance.

You can read about it here:

http://sqlcat.com/toolbox/archive/2009/02/05/a-solution-for-collecting-analysis-services-performance-data-from-many-sources-for-performance-analysis.aspx

Then download it from codeplex, here:

 http://www.codeplex.com/SQLSrvAnalysisSrvcs

A couple of days ago I needed to create a new DSV against a Teradata source. I found that the “lag” in the normally light-on-its feet DSV wizard was pretty bad - it took in the neighborhood of 1.5 minutes for the Add Related Tables function to return when I tried it against FactResellerSales (I have AdventureWorks DW loaded up in Teradata).

I was lucky to be in a LiveMeeting with some guys from Teradata who watched me struggle through this process, and they gave me some hints to make my life better – well, my data-related life, anyway:

• Use the .NET provider for Teradata version 12  (I was using the latest 8.1 Provider)
• In the data source for the project, make sure to set the database name you are connecting to in the Database property found in the All properties dialog
• In the same All properties list for the provider, set Use X Views to false

These three changes dropped my “wait time” from 1.5 minutes to 11 seconds. Can’t wait to see if queries against my ROLAP partitions come back more quickly.

Thanks Teradata guys!