So up to this point I have been pretty much been relying upon the virtual images that Microsoft creates, albeit it with tweaks and modifications of my own.  Over the course of time I have worked with many versions of the Microsoft All-Up BI demo.  The 7.1 version was a great start and the first one with which I ever worked.  I am not sure if a 8.x image was ever produced, but if it was I don’t believe it was ever made available to the general public.  The 9.2 image sucked big time and that is an understatement.  And the latest image entitled “X” (for 10 I assume) is fantastic and really shows the potential of the products and has a great deal of “eye candy” as well.  In fact, you can visit the Microsoft Online Demo Solutions site and check out 9.5 and X and judge for yourself.

Now I know Microsoft puts in a lot of effort into these and for the most part does a great job, but all these images share one fatal flaw.  Performance is problematic.  Each image is a single virtual server that has a TON of components running at the same time.  You have Windows Server, SQL Server and Analysis Services (and sometimes more than one running instance of each), Reporting Services, SharePoint Server (and all the services that go with it), Internet Information Services, the Office Suite and Internet Explorer.  It doesn’t matter if it is physical or virtual that is a huge amount to have running on one box at the same time and unfortunately performance suffers.

My laptop is a dual-core i5 2.5 GHz processor and has 8 GB RAM.  I have tried running the X running natively under Windows Server 2008 R2 using Hyper-V (I dual booted) as well as under Windows 7 using the Sun Virtual Box product.  The image ran, but I would never want to show it in a sales  or proof of concept scenario.  I have also installed the image on the Hyper-V server in our office and run it with a larger allocation of memory (12 to 20 GB) and while performance increases to acceptable levels there are still times where the system seems to get “overloaded” and it all grinds to a halt for a bit.  If I was just using this for internal learning or for my own testing and trials I wouldn’t really mind it.  However, the problem is due to the fact that prospects and clients seldom understand that the image being used in front of them and the speed at which things are happening is not typical of a production deployment.  Like it or not, this perceived slowness sticks in their mind and it can be near impossible to reverse it and when it is all said and done, who wants to buy a ”slow” solution.

So what am I doing now?  I am building my own multi-image environment.  I am lucky that TGO Consulting is a Gold Microsoft Partner and with that we are provided with certain software benefits.  Thanks to these benefits I have the freedom to build a perfect environment with the major investments just being a server to host our virtual images and some time.  Now unfortunately I have yet to figure out a way to get someone to give me a free server so I can have it all to myself so until I can I have to utilize an existing Hyper-V Server that is shared with other groups in our organization.  This means that my ideal environment is not quite there yet but at least I am getting close.

So here is the rundown of my new environment:

Image #1 – Domain Controller (Separate IP Addresses and Domain from TGO Corporate)

• 2 GB Memory (Potentially Being Lowered to 1 GB or 512 MB)
• Windows Server 2008 R2 Enterprise
• Active Directory Domain Controller

Image #2 – SQL Server

• 4 GB Memory
• Windows Server 2008 R2 Enterprise
• SQL Server 2008 R2 Enterprise
• Database Services
• Analysis Services (UDM Mode)
• Reporting Services
• Integration Services
• Adventure Works Sample DBs/Cubes
• Dynamics GP Sample DBs/Cubes

Image #3 – Application Server

• 4 GB Memory
• Windows Server 2008 R2 Enterprise
• Office SharePoint Server 2010 Enterprise (Kerberos & Claims)
• PerformancePoint Services
• Visio Services
• Excel Services
• Business Connectivity Services
• PowerPivot for SharePoint
• Etc.
• SQL Server 2008 R2 Enterprise
• Analysis Services (PowerPivot/Vertipaq Mode)

Image #4 – Windows 7 Client

• 4 GB RAM
• Windows 7 Ultimate (64-bit)
• Office Suite (64-bit)
• Excel (with PowerPivot)
• Word
• PowerPoint
• Access
• Outlook
• Visio
• Internet Explorer 9
• SharePoint Designer
• Dashboard Designer

What is really amazing about this is that in total I am consuming 14 GB of memory and I am getting far better performance than using a single image where I routinely dedicated 12 to 16 or sometimes even 20 GB of memory.  Plus when I need to do  something SQL Server intensive that does not use SharePoint or PowerPivot I can shut down the Windows 7 Client and Application Server images and temporarily assign more memory to the SQL Server.  And if we ever have a huge presentation where we need every last ounce of power from the shared Hyper-V server, I can arrange to borrow resources from other non-BI images for a while.

So what is the downside of all this?  First is that my ability to do BI presentations is directly tied to my ability to get an internet connection and make a VPN connection to our office.  So far this has not been a problem but there are organizations who believe it or not do not have internet access available to outside parties or have internet available in their meeting or boardrooms.  Second is the time it takes to make it look nice.  Although I have access to the same databases they used for the data (e.g., Adventure Works, Contoso, etc.) Microsoft did a kick-butt job on the visual aspects of the BI X image and it looks fantastic.  To replicate what they did takes a lot of time.  I have though about just doing site backup and restores from their single image to my multiple image environment and I might give that a try.

You might also ask why I am using an image for a Windows client when I could just simply connect my own laptop to this new environment and run all the client stuff locally.  It all came down to reliability and control.  On my laptop I am constantly installing widgets, add-ins, new code, whatever and sometime I blow things up.  When it comes time for an important presentation I need to be assured that everything will run perfectly.  Having an image dedicated to a Windows 7 client gives me that reliability.

The only thing left to figure out is email.  I will continue to use Outlook as the client, but what about the delivery system?  I do not want to go Exchange because 4 images is my limit of patience and it is overkill for the stuff I do.  I will probably go with the default SMTP services found within Windows Server for outbound and will look to find a freeware, shareware or open source POP service since that functionality has been removed from Windows Server 2008 R2.  I don’t know which image I will install it on and will figure that out once I find the right software.  If anyone has a suggestion please let me know.

The first announcement is that some time in the first half of 2012 is that Microsoft will enable their existing corporate BI Web-based solutions to run in browsers on Apple’s iOS and Windows devices.  I guess the assumptions here is greater support natively for the Safari browser on iOS desktop (not iPad) and support for Windows Phone 7.5.  While this is a nice announcement I don’t think it was what most of us were hoping for.  The phone offering is great but I have not yet lost a deal due to support for displaying BI data on a phone regardless of the operating system.  The iOS announcement is better news but I really wonder what it means in terms of adoption.  Most of the companies I have worked with have Apple desktop users in-house.  Mostly these are the creative folk and in these clients the IT departments have also deployed things like Citrix or Terminal Services so these Apple uses can access all the non-creative applications they need to use on a daily basis.  I think the real take away here, and what I am happy to see, is that Microsoft has now publicly recognized that there is a business opportunity to be had with publishing the end-user BI interaction to devices other than Windows desktops.

The second announcement was for the second half of 2012 and is to add touch support for data exploration and visualization capabilities of those BI offerings on iOS, Microsoft and also Android platforms.  This is the one that really intrigued me and was the one that us in the Microsoft BI world were waiting on.  Right now we are getting lots of calls by corporate executives to present BI running on the iPad (no real requests for phones or Android up to this point to be honest).  While I have made things like Excel Services and PerformancePoint Services work on an iPad with cosmetic changes via CSS the end-user experience was less than desired.  The real question is how is Microsoft going to make this happen?  Although Microsoft BI was designed with a two button mouse in mind I think the lack of right-mouse click support on the iPad and the whole using your fingers thing can be dealt with by a relatively easy revamp to the interface.  However, many  of the Microsoft BI stack components rely upon Active X  and Silverlight plug-ins.  For instance, all of Power View (once known as Project Crescent) and the PerformancePoint Decomposition Tree are Silverlight-based and these are two of the more powerful features.  The iPad Safari Browser does not allow for plug-ins of any sort nor will the upcoming native Windows 8 browser in tablet mode.  So what is Microsoft going to do?  There was no information on the technology or approach they were taking with this announcement or what specific components will or will not be made available under iOS or Android.  Hopefully after SQL 2012 ships Microsoft will be ready to speak about this more publicly.  There is a video here of Power View running on a Windows Phone, iPad and Android slate that is pretty cool but you can’t tell anything about the architecture changes they made to make it happen. I am guessing it involved migrating the current Silverlight interface code to HTML5 but that is only a guess.  I really hope that Microsoft is making the other BI items available as well because as cool as Power View is, it is not the right tool for all situations.

As a side note, the Microsoft BI team tweated a clarification today that Power View will be available with the release of SQL 2012.  However the touch specific features will come later in the year.

So on the plus side having native iOS, iPad and Android support will be awesome and it can’t get here soon enough.  However, I am worried about what will be supported and what might we give up to get that support.  The announcements are certainly a mixed-bag of emotions.

I am very happy to report that a half-year of joy, frustration, awe, anger and a dozen other emotions has finally resulted in our first release.  Below is our first semi-formal write-up for True Sky Planning, Budgeting and Forecasting.  I am sure it will undergo some marketing transformations as we refine our message, but it should give you a good overview of what we have accomplished.

* * * * * *
True Sky Planning, Budgeting and Forecasting™ is a powerful and flexible solution that helps organizations manage and streamline their planning process.  The basic philosophy of True Sky™ can best be described as:

So as I have been playing with things, I have also been reading blogs about what is happening with Windows 8.  One of the things that jumped out at me was that according to a post in the Building Windows 8 blog, if you use the Internet Explorer 10 browser that launches natively within the Metro interface you will be launching a HTML5 focused browser that prevents the loading of add-ins.  I am guessing that the reason for this is the tendency of certain plug-ins to suck up an exorbitant amount of battery and the fact that Metro is geared towards tablets, then battery is everything.  So the first thing that comes to mind for most people is support for Flash.  But I think that the success of the iPad has proven that you can get along without Flash.  

However, what also came to mind to me was the loss of Silverlight and ActiveX and what that means to Business Intelligence.  Both Silverlight and ActiveX are used quite a bit within SharePoint and the Microsoft BI stack.  For instance the Decomposition Tree and the upcoming Project Crescent are both Silverlight based and a ton of SharePoint uses ActiveX.  Luckily the engineers at Microsoft at least give us the option of switching to the “desktop” version of IE which does support plug-ins, but that transition is certainly not a seamless one or at least it isn’t right now. 

I get why Microsoft is doing this but for me personally I don’t like someone else making the decision on which plug-ins I can or cannot use.  Force me to install from an application store, give me warnings about performance and batter life up the wazzu, make me click “yes I understand” a dozen times — but let me make the choice.  If I didn’t want a choice I would buy from a guy wearing a turtleneck.

As far as SharePoint is concerned, in the current Office 365 release there is no support for the following items – PerformancePoint Services, Business Connectivity Services, FAST search or fully-trusted code.  Without PerformancePoint Services and BCS the ability to provide BI in the Microsoft public cloud is extremely limited.  It all comes down to the fact that none of these components is designed to be multi-tenant and although Microsoft is working on solutions the word I keep hearing is that we won’t see any changes until the wave 15 release of Office or beyond.   It is the “or beyond” that scares me and makes me worry how much of a priority any of these items truly have, if at all.  And when Denali is release, I really doubt we will see any sort of support for SSRS or Project Crescent within SharePoint 365 for quite a while.

Now it is true that SharePoint 365 does support Excel Services, Access Services, Visio Services and PowerPivot.  However, as I have been informed, all your data has to reside within the SharePoint environment as well.  This means any type of dashboard or scorecard or analysis you can do using these tools will be difficult as you will have to constantly be uploading Excel spreadsheets or Access databases to SharePoint in order to have updated data.  Although I suppose you could possible use the SharePoint 365 web services and create some code that resides in your on-premise environment that could automatically populate Excel with data and upload it automatically to SharePoint on a scheduled basis.  Something I will have to look into I suppose.

However, even if the auto-load with web services is possible you run into the next issue — default storage limits.  With each seat of Exchange 365 the user gets 25 GB (yes 25 GB — how cool is that!) of email storage, yet for each deployment of SharePoint 365 the organization gets 10 GB per deployment and 500 MB per user.  Isn’t there a disconnect here?  You get 50 times more storage space for email than you do for documents — it is almost as if Microsoft is saying that you should use Exchange as your document store.  With PowerPivot I regularly have workbooks that are 1 or 2 GB in size.  You can do the math on this one, but any organization serious about using PowerPivot in SharePoint 365 will most likely be paying extra very quickly.

Right now the answer from Microsoft is that you can use SharePoint 365 for your “traditional” SharePoint deployments and then do an on-premise deployment for those other components that are not supported.  What I was told (and still waiting confirmation so don’t use this as gospel) is that a user license for SharePoint 365 counts as a on-premise SharePoint CAL so you don’t have to pay twice (although you would still have to buy the SharePoint server license).  But I really can’t see most organizations splitting their deployments.  I think the most likely scenario is that they either keep it all on-premise or if they do go SharePoint 365 then they end up using some sort of other BI solution that is hosted in-house.  I could be wrong.  I guess only time will tell.

So begins my love/hate relationship with SharePoint 365.

Now I haven’t fully tested this yet, but if it works as it appears to work at first blush it opens up some opportunities in regards to data sources.  For those of you who have not worked with the BCS function of SharePoint, it allows you to connect to a data source that is SQL Server based or a .NET assembly or to a WCF web service.  Now, using a SharePoint list is not the perfect solution for a PPS data source and having to do a ”double hop” of connecting the underlying data to the SharePoint list and then the SharePoint list to PPS, but it does give us more options and more options are always better.

My big test is to see if I can find some publicly available web services and try to connect to those via BCS and then pull that in to a PerformancePoint KPI or chart/graph.  It would be very cool if you could compare actuals from your line of business system with some sort of industry or demographic data that is floating out in the cloud.

If anyone has some cool web services I might try, let me know.

• Leon Goren – President of Presidents of Enterprising Organizations and co-founder of JustWhiteShirts.com
• Joel Freedman — Chief Financial Officer of Microsoft Canada

I will be showing some pre-release code of the new TGO Budgeting Solution along with some integration to the Microsoft BI stack.  Showing beta code is always interesting so if you ever wanted to watch me crash and burn, now is the time.

For move information or to register please visit the Training & Events page on the TGO website.

In a perfect world, all data would be in a structured format (i.e., stored in databases and normalized) that would lend itself to quick, easy and powerful slicing and dicing to support the decision-making process.  Unfortunately we don’t live in that perfect world and huge amount of the data that decision makers need lies in an unstructured format.  This data resides in things like Word documents, Emails, Excel spreadsheets, PDFs and on the internet in forums and websites.  The dilemma is that compiling and analyzing this unstructured data is time consuming and difficult, but not doing so means that decisions are being made based on incomplete data sets.

Social media is all the rage right now and will continue to be so until the next big thing comes along.  An amazing amount of data on how individuals view your brand/product/service lies within sites like Facebook or Twitter.  Piss a customer off in the past and they tended to tell 10 friends about their bad experience.  Piss them off today and 2 minutes later they have posted on their wall or tweeted to 1000 people about their bad experience.  Ignore social media at your own peril.

This is where using SharePoint as a BI platform really shines. SharePoint in general and PerformancePoint Services in particular provide a platform that can easily tie together the structured data found in the line of business systems within most organizations with the ton of internally stored unstructured data that we all interact with on a daily basis.  With a few simple extra steps we can extend that integration to the unstructured social media data out on the interweb.

As a preface I want to point out that I am not a developer by trade — I am a hack.  Whatever code you see here should be taken with a grain of salt as someone who actually knows what they are doing with code can probably do it more efficiently and cleaner.  When I decided to start this project I knew I was going to use the Scorecard and thought I might use the out of the box RSS web part within SharePoint.  But search and research as I might, I found no way to pass parameters from the Scorecard to the RSS web part.  After some more searching I came across an old yet excellent blog post by Nick Barclay that got me started on a workable framework for my solution as did a post in the Microsoft PerformancePoint forum on TechNet.  Some more searching turned up the fact that Twitter has numerous exposed APIs for getting at information in different formats.  I played around with most of them before I decided I would use the Search Widget.  I did this because I liked the look and feel and because Twitter is kind enough to generate the code I needed for it.  The Search Widget is in Java so rather than pull my hair out trying to marry JavaScript and ASPX/C# (remember I am not a developer) I decided to write my page in straight HTML.  So to that end, I needed a way to pass a parameter in HTML like Nick had shown in ASPX and I found it in a post by someone who goes by the name Uzbekjon at a site called JQuery HowTo.

The final solution lets a user click on a dimension member name in a PerformancePoint dashboard and then have the results of a Twitter search appear next to it.  Jump to the bottom to see the final results or continue scrolling down to see a step-by-step description of how it all comes together:

Step 1: Create a Web Site on Your SharePoint Server to Host Your Twitter Search Results Page. Mine is http://localhost:9090/.

Step 2: Using the tool of your choice create a basic HTML page and call it “TwitterSearchResults.html”.  I am using Notepad ++ (shout out to Dan English for telling me about this tool) and it starts simple enough.

  1: <html>

  2: <!-- This PerformancePoint to Twitter Search Results Page was created by Alan Whitehouse -->

  3: <!-- http://alanwhitehouse.wordpress.com and http://www.twitter.com/alanwhitehouse --> 

  4:

  5: <head>

  6:

  7: </head>

  8:

  9: <body>

 10:

 11: <body>

 12:

 13: </html>

Step 3:  Add the code to extract the parameters that are within the URL that our page will be receiving from the PerformancePoint scorecard.  To note is line #20.  The syntax here caused me some headache before I figured out what I should put.  I kept using “EndPoint_URL” which is what you see in the technical documentation but in reality what was coming across was “EndPoint%5FURL” where “%5F” is the translation for the underscore character.

  1: <html>

  2: <!-- This PerformancePoint to Twitter Search Results Page was created by Alan Whitehouse -->

  3: <!-- http://alanwhitehouse.wordpress.com and http://www.twitter.com/alanwhitehouse --> 

  4:

  5: <head>

  6: <script language="JavaScript" type="text/javascript">

  7: function getUrlVars()

  8: {

  9: var vars = [], hash;

 10: var hashes = window.location.href.slice(window.location.href.indexOf('?') + 1).split('&');

 11: for(var i = 0; i < hashes.length; i++)

 12: {

 13: hash = hashes[i].split('=');

 14: vars.push(hash[0]);

 15: vars[hash[0]] = hash[1];

 16: }

 17: return vars;

 18: }

 19:

 20: var SearchTerm = getUrlVars()["EndPoint%5FURL"];

 21: </script>

 22: </head>

 23:

 24: <body>

 25:

 26: <body>

 27:

 28: </html>

Step 4:  Add the Search Widget code from Twitter to your HTML page.  The easiest way to get this code is to visit the Twitter site I mention above and use the tool they have there to generate the code for you.  By design, the tool looks to hard-code the search parameters and if you look at line #30 below, you will see that I have replaced the default value with my variable that I created in line #20.  Save your finished HTML page to the web site you created in Step 1 above.

  1: <html>

  2: <!-- This PerformancePoint to Twitter Search Results Page was created by Alan Whitehouse -->

  3: <!-- http://alanwhitehouse.wordpress.com and http://www.twitter.com/alanwhitehouse --> 

  4:

  5: <head>

  6: <script language="JavaScript" type="text/javascript">

  7: function getUrlVars()

  8: {

  9: var vars = [], hash;

 10: var hashes = window.location.href.slice(window.location.href.indexOf('?') + 1).split('&');

 11: for(var i = 0; i < hashes.length; i++)

 12: {

 13: hash = hashes[i].split('=');

 14: vars.push(hash[0]);

 15: vars[hash[0]] = hash[1];

 16: }

 17: return vars;

 18: }

 19:

 20: var SearchTerm = getUrlVars()["EndPoint%5FURL"];

 21: </script>

 22: </head>

 23:

 24: <body>

25:  <script src="http://widgets.twimg.com/j/2/widget.js"></script>

 26: <script>

 27: new TWTR.Widget({

 28:   version: 2,

 29:   type: 'search',

 30:   search: SearchTerm,

 31:   interval: 1000,

 32:   title: '',

 33:   subject: '',

 34:   width: 'auto',

 35:   height: 300,

 36:   theme: {

 37:     shell: {

 38:       background: '#ffffff',

 39:       color: '#ffffff'

 40:     },

 41:     tweets: {

 42:       background: '#ffffff',

 43:       color: '#444444',

 44:       links: '#1985b5'

 45:     }

 46:   },

 47:   features: {

 48:     scrollbar: true,

 49:     loop: false,

 50:     live: true,

 51:     hashtags: true,

 52:     timestamp: true,

 53:     avatars: true,

 54:     toptweets: true,

 55:     behavior: 'all'

 56:   }

 57: }).render().start();

 58: </script>

 59: <body>

 60:

 61: </html>

Step 5:  Go into Dashboard Designer and create a new Web Page Report.  Point the URL to your website location where you have stored the “TwitterSearchResults.html” web page we designed above.

Step 6: Create the KPI you want to use on the Scorecard.

Step 7: Create the Scorecard you want to use in the Dashboard.

Step 8: Pull your Scorecard and Web Page Report into a Dashboard and connect the Scorecard to our Twitter Search Web Page Report.

Step 9: Deploy your Dashboard and view the results.  You will notice that whatever row name you click on becomes a search term for Twitter.  In addition, if you click (no image shown) on one of the Tweets the Twitter JavaScript code will actually allow you to interact with the Tweet and reply or retweet it or to add the person to your follow list.

Obviously there are additional formatting and cosmetic changes that can also be done but what I like about this solution is that you can attach the Twitter Search Results report to pretty much any Scorecard and it will work without having to change a line of your HTML code.  If you can streamline my code or process I would love some suggestions.  Also, if anyone has a good suggestion for posting code examples into WordPress that format nicely and is quick to use please let me know.

1. Exactly what products are you talking about when you refer to the Microsoft BI stack?
2. Where do I get more information on a particular product in the stack?

So being the lazy and generally unsociable person I am, I decided to bite the bullet and create a post that I could just point people to rather than having to retype my answers over and over again.  So below are the components that I consider part of the Microsoft BI solution offering, as well as some links to more information on those products.  If I have missed anything or if you have some suggestions for some additional/better links drop me a line and I will add it to the list.  I am also going to pin this to my main navigation bar so people can find it easily.  So without further adieu here is the list:

Lasted Updated: October 13, 2011

UPDATE 1:  The image finally downloaded and we got it all unzipped.  It is running on a Hyper-V server in my office and I finally tried to access it this morning via my hotel internet connection (I am on the road right now).  Didn’t have time to play with it much as the hotel connection was running at about 14.4 baud modem speed but the  opening landing page is total eye candy and will be great for first impressions.  No idea how good the data is yet, hopefully will get some time tomorrow to really dive deep.

• MM/DD/YYYY
• DD/MM/YYYY
• MM/DD/YY
• DD/MM/YY

On top of that I also had:

• Empty Date Fields
• Date Fields with Invalid Dates (i.e., 00 for the Month or Day)
• Date Fields with Impossible Dates (i.e., 2601 for the Year)

Althought my imports worked fine, I found out that when you bring in dates in a mix of formats or with invalid values if you try to format the date column as a date PowerPivot errors out.  It is understandable if you think about it.  If you had 08/12/2010 and 12/08/2010 then your day and month might be wrong but the system could deal with it.  What it can’t seem to deal with is something like 03/28/2010 and 28/03/2010.  I also found out it definitely can’t deal with blank dates or months/days that are 00.

So my colleague Richard Mintz suggested we might try a CASE statement to put some logic into the query to adjust for the different formats.  But no matter what we tried the CASE statement kept erroring out.  Richard’s thought is that because of the fact that the process we had to use to merge the different Excel sheets utilizes the Access/Jet driver and Access does not support CASE this was the problem.   If anyone can confirm or deny this then I would love to hear from you.

In this whole process I am trying to avoid manually manipulating the source files each time we get a new data dump.  And since I could not deal with this issue in the query that meant dealing with it within the PowerPivot end-user environment.  As a reminder here is  a portion of my original query:

SELECT ['MasterData$'].*
FROM ['MasterData$']
UNION ALL
SELECT [DataSource1$].XX_ID as ‘UniqueID’,
[DataSource1$].TITLE as ‘Salutation’,
[DataSource1$].FNAME as ‘First Name’,
[DataSource1$].LNAME as ‘Last Name’,
” as ‘Title’,
” as ‘Company ID’,
[DataSource1$].COMPANY as ‘Company Name’,
[DataSource1$].ADDR1 as ‘Address 1′,
[DataSource1$].ADDR2 as ‘Address 2′,
[DataSource1$].CITY as ‘City’,
[DataSource1$].PROVINCE as ‘Province’,
[DataSource1$].POSTAL as ‘Postal Code’,
” as ‘Phone’,
[DataSource1$].LAST_CONTACT_DATE as ‘Imported Date’,
‘M/D/Y’ as ‘Imported Date Format’,
[DataSource1$].SALE_AMOUNT as ‘Sale Amount’,
‘DataSource1′ as ‘Source’
FROM C:\file_path\UglyDataSource1.xlsx.[DataSource1$]
UNION ALL (repeat with next data source and modified as necessary)……

Highlighted in red are two lines around the date issue that may seem a little strange.  I brought in the dates into a field I called “Imported Date” and with each data source I flagged the date format using the “Imported Date Format” field.  Lets shift gears and move to the PowerPivot environment to complete the work.   Take all I do in PowerPivot with a grain of salt as I am a novice at PowerPivot and this whole problem could possibly have been handled much easier than my approach. 

The screenshots show the whole process already completed, we will be working from left to right.  In the first screenshot below you can see how the raw dates come into the system.  I know that everything in this picture seems to be the same, but that is due to the fact that you are only seeing data from the first import source.  If you were to scroll down you would see the mess I am dealing with in my scenario.  So after I brought my data in the first thing I did was to highlight my column and change the data type from Text to Date.

 However when I tried that, I got the following lovely error:

So begins the process to fix my data.   The first thing I did was to deal with my dates that were blank.  For those I decided I would assign them a date that was so outrageous that everyone would immediately recognize that this was a plug.  For this example I used “12/31/9999″.   It is nice to know that PowerPivot has some serious life left in it in terms of the dates it can handle.

The next two steps were to deal with the situation where I had months or days that were stored as “00″.  For these I decided to change the value to “01″.  I don’t have the option of doing something like I did for the year with “9999″ so this was the next best thing.

After that I had to deal with the fact that the year was sometimes coming in as 2 digits (MM/DD/YY) and sometimes as 4 digits (MM/DD/YYYY).

After that fix, the final step was to bring it all together.  This was where I was able to adjust for the MM/DD/YYYY vs. DD/MM/YYYY issue and finally turn my text column into a date column.

I like doing my cleanup in steps in different columns as it is easier for me to troubleshoot, but there is nothing keeping you from doing this all in one step in one really big Excel function.  Again, if anyone has a better suggestion for how to deal with this issue, I would love to hear from you.

So my first challenge was how to get these spreadsheets into PowerPivot.  Now PowerPivot can easily bring in tables from Excel, but it is designed to bring in different data that can be logically linked.  My need was to consolidate all these data sources into a single table.  After pulling my hair out on how to do this, my colleague Richard Mintz (he is blogging now, check him out) pointed me to a blog post that helped me figure out what I needed to do.  Now this post assumed all the files were the same so I took his method and modified it somewhat as detailed below:

1. I created a blank “master” Excel workbook that contained the fields I wanted to populate with “user friendly” names rather than start with one of the data sources first.
2. I created my data connections to all my data source spreadsheets.
3. I brought in my master Excel workbook first and then began modifying the import query for the rest of the imports.

It was in step #3 where I began to forge my own way.  So after the “UNION ALL” statement, instead of a “Select * From C:\\file_path\datasource.xlsx.[TabName$]“  statement I modified it to be a “Select As” type statement where I mapped my raw field names to my user friendly names.   An example is below:   

SELECT ['MasterData$'].*
FROM ['MasterData$']
UNION ALL
SELECT [DataSource1$].XX_ID as ‘UniqueID’,
[DataSource1$].TITLE as ‘Salutation’,
[DataSource1$].FNAME as ‘First Name’,
[DataSource1$].LNAME as ‘Last Name’,
” as ‘Title’,
” as ‘Company ID’,
[DataSource1$].COMPANY as ‘Company Name’,
[DataSource1$].ADDR1 as ‘Address 1′,
[DataSource1$].ADDR2 as ‘Address 2′,
[DataSource1$].CITY as ‘City’,
[DataSource1$].PROVINCE as ‘Province’,
[DataSource1$].POSTAL as ‘Postal Code’,
” as ‘Phone’,
[DataSource1$].LAST_CONTACT_DATE as ‘Imported Date’,
‘M/D/Y’ as ‘Imported Date Format’,
[DataSource1$].SALE_AMOUNT as ‘Sale Amount’,
‘DataSource1′ as ‘Source’
FROM C:\file_path\UglyDataSource1.xlsx.[DataSource1$]
UNION ALL (repeat with next data source and modified as necessary)……

 So a couple of things I learned by doing this whole process:

• It can be done but it is not the easiest or most intuitive thing to do.
• You have to map to every field in your master document.  If a data source does not have that field, then you have fill it in with a blank or “n/a” or some other value.  This is what happened to me for in the sample above Title, Company ID and Phone.   Each of my queries had to be tweaked individually.
• If you are bringing in data from multiple sources I found it helpful to add a column to track where the data came from hence the field “Source” that I added.
• Hand coding the complete query in the PowerPivot window was tedious at best.   I gave up and wrote it in Notepad and then copied it over.
• My different data sources all treated dates differently and this was very problematic.   However that is the subject of my next post.

If anyone has an easier way to do this I would love to hear about it.