Long HTTP Get and Instant Response

Question

• We have an ASP.Net 3.5 Application on IIS7.

• On our preview server, it runs with almost instant GETs and responses (~1 second from get to the page displaying).

• We have a production environment with 5 web front ends and a cluster of 3 SQL databases.

• The font ends are Load Balanced with BigIP from F5 on round robin.

• There is a firewall between the front ends and the SQL servers.

In production, we are experiencing a 45 second Get before the server starts its response. Based on the information above, what could the culprit be?

Answer 1

Any of the components you mentioned could be the cause of the problem.

The best way to troubleshoot the issue would be to sniff the inbound and outbound interfaces on a client, the IIS servers and the SQL servers. You should have no trouble determining the source of your problem once you have a record of where your request, and its subsequent response, are during the 45 second window.

Answer 2

I agree with @Rich, the problem could be anywhere. To be specific, use a logging library for your dotNet app and check that, check your IIS logs, and maybe use SQL Server Profiler on your database servers.

The only complexity is the clusters, so you'll only see 1/5 of the web traffic and 1/3 of the database traffic. If every request is 45 seconds slow, then I guess tracing any request will tell you the problem. Otherwise, turn on sticky sessions so you always hit the same server for your entire testing session.

Of course, we're assuming you've checked the basics - your servers aren't starved for resources, right? (i.e.: cpu not maxed, still have free RAM, page file not maxed.)

Answer 3

Also, if you are re-checking the basic make sure you verify that all of your networked devices have the correct speed and duplex settings. It's amazing how often a single server with an interface that is set to half duplex will ruin an otherwise functional application.

Answer 4

Do you have monitoring on the f5 setup? If you enable the default monitoring it checks every 5 seconds and would mark the server offline after 12 - So you would know if the problem was between the F5 and the web servers. If you don't have basic monitoring you might want to consider setting it up so that the F5 can make intelligent decisions about the nodes