Troubleshooting event id 5146 "Application pool has been disabled" on IIS7

Question

I have a website running on Windows Server 2008 Web Edition, that has suddenly in the last few days started freezing and giving users a "503 Service Unavailable" message.

There's a bunch of errors showing up in the event log, but the most interesting one is:

 **Log Name: System Source: Microsoft-Windows-WAS Date: 22/08/2011 12:11:16 Event ID: 5146 Task Category: None Level: Error Keywords: Classic User: N/A Computer: myserver.mydomain.com Description: Application pool www.mysite.com has been disabled. The request from protocol http to create the application pool failed. Restart the application pool so that Windows Process Activation Service (WAS) can determine the correct state of the protocol. The data field contains the error number. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-WAS" Guid="{524B5D04-133C-4A62-8362-64E8EDB9CE40}" EventSourceName="WAS" /> <EventID Qualifiers="49152">5146</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2011-08-22T11:11:16.000Z" /> <EventRecordID>37</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>myserver.mydomain.com</Computer> <Security /> </System> <EventData> <Data Name="AppPoolID">www.mysite.com</Data> <Data Name="ProtocolID">http</Data> <Binary>B7000780</Binary> </EventData> </Event> 

A server reboot fixes the problem temporarily - for ca. 24 hours - after which we start seeing 503 error messages again. There's nothing much to go on in the IIS event logs; restarting the application pool doesn't fix it, neither does an iisreset.

EDIT: The site is now failing immediately, and we still have no idea what's causing it.

Any ideas?

Thanks

-D-

Answer 1

If a reboot is the fix, troubleshoot the problem as a likely memory leak.

Answer 2

Thanks to Microsoft support, we've tracked this down to some sort of resource leak related to Windows file sharing. The web servers in question have a virtual directory mapped to a UNC file share on a media server, and - for reasons we don't entirely understand yet - file handles opened on this UNC share weren't being closed cleanly.

MS support analysed the memory dump from the crashed IIS process, observed that it was littered with references to UNC file paths on \contentserver\webmedia\, and advised us to reboot the media server... we did this yesterday morning and so far, everything's running normally. We've investigating a potential problem with the firewall appliance that sits between the web servers and content server, but if you're experiencing something similar and your website's got virtual directories mapped to UNC shares, try restarting the servers on the other end of the UNC shares.