So you want to be on the latest and greatest software with your Business Intelligence platform…well watch out for this one. Back in January this year I was in the middle of a major SQL Server 2008 and Windows Server 2008 upgrade when we started to experience some issues with our Reporting Services (SSRS) 2008 testing against Analysis Services (SSAS) 2008 data sources. What we were doing was building out all of the SQL Server 2008 components with Windows Server 2008 and moving the SQL Server 2005 items to these new servers to perform testing. All of the SQL Server 2005 components were running on Windows Server 2003.
The error that we were running into when trying to running SSRS 2008 reports against SSAS 2008 was the following:
The connection either timed out or was lost. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. An existing connection was forcibly closed by the remote host
So after getting this error we made sure that Kerberos was properly configured and then started to dive into a more deeper analysis using WireShark and reviewing network traces. We also made sure that all of the servers had updated firmware and drivers for the network adapters (http://support.microsoft.com/kb/942861) and we turned off all offloading on the adapters to see if that would resolve the issue including the TCP Chimney (sorry Santa, we closed the Chimney down
) and UPD Checksum and any other Offloading that was setup on the network adapters (http://support.microsoft.com/default.aspx/kb/951037) – these properties can be found if you go into Device Manager—>Network adapters—>and check the Advanced tab of the properties of each network adapters.
The funny thing was that if we pointed SSRS 2008 against our SSAS 2005 server (that was running Windows Server 2003) the reports executed just fine. We also ran all of the reports with issues from SSRS 2005 servers to the SSAS 2008 servers and they worked fine. So what was going on? After numerous testing and network traces we finally determined that this was out of our control and needed to contact Microsoft.
After going through more testing, network traces, SSRS log files, etc. and about seven support people I finally got a phone call one night while I was in Cub Foods getting a few groceries at about 8 or 8:30 at night. The Microsoft engineer was with the SSAS team and he was aware of the issue…this was starting to pop-up with customers and we were about the fourth case of this that he had heard of
After additional testing it turns out there is an issue with AES (advanced encryption standard) aware operating systems (Vista, Windows Server 2008, Windows 7) and with Kerberos. I was given a very detailed description of different messages that are used with Kerberos. There is an issue with the decrypt data for security buffer not being the correct size for AES and this causes the error. This occurs when you are using Kerberos and are trying to get AES aware operating systems to talk to each other.
So our workaround was to keep the front-end servers (SSRS, PerformancePoint, ProClarity, and SharePoint) on Windows Server 2003 and the backend servers for our databases on Windows Server 2008. This was a very frustrating item to uncover and took us a few weeks of testing along with additional time working with Microsoft on the issue. The Microsoft case has been moved over to the Windows Server team and they have stated that a fix should hopefully be available in May 2009 (need to perform a few months of testing and going through the red tape for the hotfix).
For more information in regards to this and a similar posting check out this link – SSAS: Kerberos kills ‘large’ MDX queries on Windows Server 2008. Sorry I didn’t get this posted sooner to warn more people, but hopefully this will now help out other people experiencing this same issue or planning their infrastructure upgrades
UPDATE (4/5/2009): I saw that one of the engineers on my case posted a blog posting here in reference to this issue if you want some more details – Errors may occur after configuring Analysis Services to use Kerberos authentication on Advanced Encryption Standard Aware Operating Systems.
UPDATE (6/2/2009): The knowledge base number assigned to this is 968700, but has not been published yet since a hotfix is not available at this time. Based on Microsoft information they now state that this will possibly slip to August for the fix, but there is a possibility of a June (or July) release if it passes all of the comprehensive testing that they require. They did state that this is resolved with Windows 7 and also Windows Server 2008 R2. Unfortunately these operating systems are in Release Candidate right now and are slated to be released possibly in Q3 of this year. So the wait continues
Keep using Windows Server 2003 for either the front-end servers or the database (front-end probably easier, especially if they are virtual — just swap them out).
UPDATE (7/20/2009): According to Dr. John Tunnicliffe a patch has been released for this and is available to download – SSAS: Microsoft release fix for “Kerberos killing MDX” issue .
Microsoft has released a patch to us that fixes the issue. However, we do not know when Microsoft intends to roll this patch into a service pack for Windows Vista/Server 2008. As several people have contacted me about the issue, I provide the patch for those suffering the same plight can get a quick fix. There is no warranty and no guarantee with this patch. Works for us, but test it first!
UPDATE (8/23/2009): Removed old links and replaced with the official KB link here –> http://support.microsoft.com/default.aspx/kb/969083
Subscribe to Blog feed
Dan English's BI Blog 