Dan English's BI Blog

Welcome to my random thoughts in regards to Business Intelligence, databases, and other technologies

Windows Server 2008 Kerberos Bug – Transport Connection Issues with SSAS data

Posted by denglishbi on March 31, 2009

So you want to be on the latest and greatest software with your Business Intelligence platform…well watch out for this one.  Back in January this year I was in the middle of a major SQL Server 2008 and Windows Server 2008 upgrade when we started to experience some issues with our Reporting Services (SSRS) 2008 testing against Analysis Services (SSAS) 2008 data sources.  What we were doing was building out all of the SQL Server 2008 components with Windows Server 2008 and moving the SQL Server 2005 items to these new servers to perform testing.  All of the SQL Server 2005 components were running on Windows Server 2003.

The error that we were running into when trying to running SSRS 2008 reports against SSAS 2008 was the following:

The connection either timed out or was lost. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. An existing connection was forcibly closed by the remote host

So after getting this error we made sure that Kerberos was properly configured and then started to dive into a more deeper analysis using WireShark and reviewing network traces.  We also made sure that all of the servers had updated firmware and drivers for the network adapters (http://support.microsoft.com/kb/942861) and we turned off all offloading on the adapters to see if that would resolve the issue including the TCP Chimney (sorry Santa, we closed the Chimney downCrying) and UPD Checksum and any other Offloading that was setup on the network adapters (http://support.microsoft.com/default.aspx/kb/951037) – these properties can be found if you go into Device Manager—>Network adapters—>and check the Advanced tab of the properties of each network adapters.

The funny thing was that if we pointed SSRS 2008 against our SSAS 2005 server (that was running Windows Server 2003) the reports executed just fine.  We also ran all of the reports with issues from SSRS 2005 servers to the SSAS 2008 servers and they worked fine.  So what was going on?  After numerous testing and network traces we finally determined that this was out of our control and needed to contact Microsoft.

After going through more testing, network traces, SSRS log files, etc. and about seven support people I finally got a phone call one night while I was in Cub Foods getting a few groceries at about 8 or 8:30 at night.  The Microsoft engineer was with the SSAS team and he was aware of the issue…this was starting to pop-up with customers and we were about the fourth case of this that he had heard ofEye-rolling

After additional testing it turns out there is an issue with AES (advanced encryption standard) aware operating systems (Vista, Windows Server 2008, Windows 7) and with Kerberos.  I was given a very detailed description of different messages that are used with Kerberos.  There is an issue with the decrypt data for security buffer not being the correct size for AES and this causes the error.  This occurs when you are using Kerberos and are trying to get AES aware operating systems to talk to each other.

So our workaround was to keep the front-end servers (SSRS, PerformancePoint, ProClarity, and SharePoint) on Windows Server 2003 and the backend servers for our databases on Windows Server 2008.  This was a very frustrating item to uncover and took us a few weeks of testing along with additional time working with Microsoft on the issue.  The Microsoft case has been moved over to the Windows Server team and they have stated that a fix should hopefully be available in May 2009 (need to perform a few months of testing and going through the red tape for the hotfix).

For more information in regards to this and a similar posting check out this link –  SSAS: Kerberos kills ‘large’ MDX queries on Windows Server 2008.  Sorry I didn’t get this posted sooner to warn more people, but hopefully this will now help out other people experiencing this same issue or planning their infrastructure upgradesNerd

UPDATE (4/5/2009):  I saw that one of the engineers on my case posted a blog posting here in reference to this issue if you want some more details – Errors may occur after configuring Analysis Services to use Kerberos authentication on Advanced Encryption Standard Aware Operating Systems.

UPDATE (6/2/2009): The knowledge base number assigned to this is 968700, but has not been published yet since a hotfix is not available at this time.  Based on Microsoft information they now state that this will possibly slip to August for the fix, but there is a possibility of a June (or July) release if it passes all of the comprehensive testing that they require.  They did state that this is resolved with Windows 7 and also Windows Server 2008 R2.  Unfortunately these operating systems are in Release Candidate right now and are slated to be released possibly in Q3 of this year.  So the wait continuesSad  Keep using Windows Server 2003 for either the front-end servers or the database (front-end probably easier, especially if they are virtual — just swap them out).

UPDATE (7/20/2009): According to  Dr. John Tunnicliffe a patch has been released for this and is available to download – SSAS: Microsoft release fix for “Kerberos killing MDX” issue .

Microsoft has released a patch to us that fixes the issue. However, we do not know when Microsoft intends to roll this patch into a service pack for Windows Vista/Server 2008. As several people have contacted me about the issue, I provide the patch for those suffering the same plight can get a quick fix. There is no warranty and no guarantee with this patch. Works for us, but test it first!

UPDATE (8/23/2009): Removed old links and replaced with the official KB link here –> http://support.microsoft.com/default.aspx/kb/969083 

Advertisements

6 Responses to “Windows Server 2008 Kerberos Bug – Transport Connection Issues with SSAS data”

  1. Colleen said

    Thanks for blogging this – this issue has caused us much frustration. I spent most of January trying to figure out what the problem was trying to connect to our cubes. We had set up all our new shiny BI stuff (sql server 2008, performance point 2007, sharepoint 2007) on windows server 2008. I submitted an issue with microsoft on 1/19/09 and worked with a tech for several weeks of experimentation – by feb 19 they confirmed that it was an issue with "Kerberos authentication fails when using AES aware operating systems". We moved Performance point back to 2003 but we’ve been holding out to move Sharepoint back in hopes that the fix would come out sooner rather than later. May have to give up if the hotfix doesn’t come out till August.

  2. Michael Grimm said

    Dan, the patch worked great for Zack and I. Thank you so much!!!

  3. Dan said

    Awesome and glad to hear:) Hope all is well. You can contact me at my work email since I currently do not have access to my Microsoft email account.

  4. Unknown said

    Is it possible that this problem also applies to SSRS 2008 in sharepoint 2007 integration connecting to sharepoint lists via XML Connection type? – Feb 2010

  5. Dan said

    This was strictly a Kerberos related bug with Vista and Windows Server 2008. It was fixed with this patch and has been resolved in Windows Server 2008 R2 (and Windows 7).

  6. Sam Kane said

    Here are this and some other articles on Analysis Services and Kerberos:

    http://ssas-wiki.com/w/Articles#Kerberos

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: