Thursday, April 16, 2015

Troubleshooting MSDTC when vCenter Storage & Network is not detected by vRealize Automation

Welcome: To stay updated with all my Blog posts follow me on Twitter @arunpande !

I am currently working on vRealize Automation 6.2 implementation where I have completed the distributed install. When. Post the installation I started the basic configuration where I added the vCenter server, Fabric & Business Groups.
However when I was creating the reservations I noticed that the Storage & Network details were not detected. This was not new, I had faced this issue earlier but it was not an easy fix this time as the Windows firewall was enabled on the Windows database & IaaS servers and it could not be disabled.
In this blog post I would like to share the different troubleshooting steps that I have performed to troubleshoot and fix this issue.
Step 1 – Have a clear understanding about the problem statement.
In this case the Storage Paths and Network was not detected when creating the Reservations for Business Groups.
Note that at least one data collection should be completed successfully for the Compute Resource before this data is populated in the Reservations.
Step 2 – Investigating the cause & FIX the issue
Look at the status of the vSphere Endpoint make sure that it’s OK. To confirm this, navigate to Infrastructure > Compute Resources > Compute Resource.
Next navigate to Infrastructure > Monitoring > Log to check the errors. In this case the below errors were reported.
Error
2/4/2015 7:55 PM
Manager Service
Manager Service
XXXXX
XXXXX
DataBaseStatsService: ignoring exception: Error executing query usp_SelectAgent Inner Exception: Error executing query usp_SelectAgentCapabilities
https://mycloud.vsnl.co.in/vcac/Images/icon_error.pngError
2/4/2015 7:55 PM
Manager Service
Manager Service
XXXXX
XXXXX
Error processing ping response Error executing query usp_SelectAgent Inner Exception: Error executing query usp_SelectAgentCapabilities
https://mycloud.vsnl.co.in/vcac/Images/icon_error.pngError
2/4/2015 7:54 PM
Manager Service
Manager Service
XXXXX
XXXXX
DataBaseStatsService: ignoring exception: Error executing query usp_SelectAgent Inner Exception: Error executing query usp_SelectAgentCapabilities
https://mycloud.vsnl.co.in/vcac/Images/icon_error.pngError
2/4/2015 7:54 PM
Manager Service
Manager Service
XXXXX
XXXXX
Error processing ping response Error executing query usp_SelectAgent Inner Exception: Error executing query usp_SelectAgentCapabilities


While the above errors indicate a possible issue with the IaaS database for detailed information, check the Manager Service logs that is located in C:\Program Files (x86)\VMware\vCAC\Server\Logs.
NOTE – If you have multiple servers with manager service installed, check the logs in the server which is Active. You can either check this from the Load balancer to check which node is Active or you can log into the Server and check the status of the below service, it would be running on the Active server.
Here is the snip of the errors reported in the ALL.txt file
System.ApplicationException: Error executing query usp_SelectManagementEndpoint  ---> System.ApplicationException: Error executing query usp_SelectEntityProperties  ---> System.Transactions.TransactionManagerCommunicationException: Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool. ---> System.Runtime.InteropServices.COMException: The transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D024)
  at System.Transactions.Oletx.IDtcProxyShimFactory.ReceiveTransaction(UInt32 propgationTokenSize, Byte[] propgationToken, IntPtr managedIdentifier, Guid& transactionIdentifier, OletxTransactionIsolationLevel& isolationLevel, ITransactionShim& transactionShim)
  at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)
  --- End of inner exception stack trace ---


This clearly indicates that the MSDTC between the SQL Server & the Web Server was not working. However the MSDTC settings have been configured as per the vRA documentation:
The next thing I tried to do was a DTCPing from the Database Server to the Web Server. DTCPing is a tool provided by Microsoft to troubleshoot MSDTC. The tool is available for free download from http://www.microsoft.com/en-in/download/details.aspx?id=2868.
Once you have downloaded DTCPing.exe, run the installer and extract the files in a folder.



You will now see the following files in the folder after extracting the executable.

Repeat the same steps on the Web Server.
Launch the Dtcping on both servers
Sine the communication is initiated from the Web Server to the DB server enter DB server hostname in the MSDTC Simulation window launched on the Web Server
Check the status of the DTCPing command. In my case it initially received an error “RPC Server is unavailable”.  To fix the MSDTC issue, I performed the following steps:
Make sure that the MSDTC is enabled between all the servers so if you have a distributed install of vRA and have multiple Web Servers ensure that MSDTC is enabled.
There shouldn’t be any firewalls between this servers, in case there is a firewall and make sure that the MSDTC ports are open.
IMPORTANT – Disable the Windows firewall on the Web Servers & SQL Database. If this is against customer compliance and security policies you can enable the Windows firewall but create appropriate rules which allows the communication between the two host.
Launch the Windows firewall and click on Advanced Settings > Click on Inbound Rules and click on New Rule.
In Rule Type select Program
Enter the complete path of msdtc.exe
HINT – To find the location, open the task manager and click in Details
Right click on the msdtc.exe application and click on Properties
In Properties window for msdtc.exe you will find the
Now back to our firewall rule. Enter the correct path the msdtc.exe application and click Next
Click on Allow the connection and click Next
I chose the default options
Enter a name and description of the rule and click Finish to create this rule.
Login to the server with Manager Service installed. NOTE: In distributed install, Manager Service is installed in Active/Passive mode, to check the Active node login to the Load balancer.
Restart the below service
To confirm the status, check the server logs located at C:\Program Files (x86)\VMware\vCAC\Server\Logs.
If the issue persists, perform the below steps on both the Database & IaaS Servers
  • Uninstall MSDTC from Windows Command prompt using msdtc –uninstall command.
  • Reboot the Manager Service server
  • Install MSDTC on the server using command msdtc –install from the command prompt
  • Reboot the Manager Service server
  • Make sure MSDTC is enabled as per the below screenshot

UPDATE - set the MSDTC service startup type to Automatic after reinstalling, as it defaults to manual:
sc config msdtc start= auto
sc start msdtc