Changes in Exchange 2010 SP 3 that could affect companies that have a large Outlook Anywhere population
Exchange 2010 SP3 was released recently. The big fanfare was that it allows for co-existence between Exchange 2010 and Exchange 2013. This is something that is routine when a new Exchange product is released, as the old product needs to be updated to know that there is a new product out there. This usually takes the form of Schema Updates and its pretty benign.
One thing that was done in service pack 3 for Exchange 2010 can cause an issue for companies who heavily use Outlook Anywhere. These companies may start to see users calling saying that they either cannot connect or are losing connectivity to their mailboxes. I plan to address this particular issue in this post.
In service pack 3, Exchange for the first time is going to utilize the ASP.NET runtime to process RPC over HTTP connections (Outlook Anywhere connections). ASP.NET runtime environment was architected according to design principles that place a premium on reliability and performance. The resultant ASP.NET process model comprises two system elements—an in-process connector living inside the Web server process and an external worker process. In addition, the ASP.NET runtime infrastructure is scalable enough to automatically exploit any selected processor on multi-processor hardware.
ASP.NET has a connection limit of 25,000, by default. In some organizations, this may not be high enough. Once this limit is reached, users start getting dropped or they cannot connect at all. Think about it this way, once your Exchange server reaches the RPC Request limit of 500, users are disconnected or cannot connect, making for a poor user experience. This is a similar issue, just with a different component, ASP.NET.
There are a couple of ways of determining if you have hit this limit. The first is to fire up Performance Monitor, go to the object ASP.NET Apps V2.0.50727, expand that out and choose Requests Executing and choose the counter _LM_W3SVC_1_ROOT_Rpc and under the RPC/HTTP Proxy object add the counter "Current Number of Incoming RPC over HTTP Connections". Both these counters should be below 25,000. If you are close, watch the counter. If you see that the counter bumps to 25,001 and then drops to 24,994 you are exceeding this maximum. Below is what this looks like for the "Current Incoming of Incoming RPC over HTTP Connections":
These 2 counters track the same exact way, so having only one or the other will be sufficient. I had to choose "Scale the selected counter" by Right Clicking on the counter.
The second way to see if you are running into this issue is to look in the IIS logs. You will see the following entries:
2013-03-21 23:06:14 10.0.0.1 RPC_OUT_DATA /rpc/rpcproxy.dll cas.contoso.com:6001 443 CONTOSO\user 10.0.0.10 MSRPC 503 3 0 15
2013-03-21 23:06:15 10.0.0.1 RPC_IN_DATA /rpc/rpcproxy.dll cas.contoso.com:6001 443 CONTOSO\user 10.0.0.10 MSRPC 503 3 0 46
These 2 items are clear indications that you are hitting this limit.
The good news is there is a simple fix for this. The following steps can be used to increase this limit:
- Verify which app pool the /RPC virtual directory is using appcmd:
- cd %windir%\system32\inetsrv
- appcmd list app /path
/RPC this will show which app pool for example
APP "Default Web Site/Rpc" (applicationPool:DefaultAppPool)
2. Verify the .net Framework the app pool is using for example:
appcmd list apppool /name:DefaultAppPool
APPPOOL "DefaultAppPool" (MgdVersion:v2.0,MgdMode:Integrated,state:Started)
In the above Example we know the RPC\HTTP is using .net v2.0 managed framework (this is the default) and would use the
following instructions:
3. On the computer that is running the Client Access server role, locate the following file:
C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\CONFIG\Machine.config
4. Open the Machine.config file.
5. In the <system.web> section, locate the following line:
<processModel
requestQueueLimit="25000" autoConfig="true" />
6. Change the value of requestQueueLimit property to 65535.
7. Open a command prompt and type IISReset /noforce **
**Please note that if you have other applications running on your Exchange 2010 SP3 server that utilize IIS, doing an IISReset /noforce will also restart those applications, as well as your Exchange web applications.**
These recommendations are noted in Knowledge Base Article 2832198
How do you proactively get ahead of this? That's a good question. It is pretty simple though. I recommend that you start to look at the counters in Performance Monitor that are listed above. These counters are available for you to monitor in Exchange 2010 SP2. I would look at these counters over a span of a week or so to get an idea of what their daily average is. If they are well under 25,000 there will be no need to make the above changes to the machine.confg file. If you are close or going over the 25,000 limit that is imposed by ASP.NET you will need to make the changes to the machine.config file after applying SP3.
I would like to recognize Scott Oseychik, Bryan Talbert and Lou Prete for their assistance on this blog. Their knowledge and insight into Exchange and IIS, respectively, have been invaluable.