Have a look here first: http://stackoverflow.com/questions/5349867/error-handling-in-asp-net-mvc3https://stackoverflow.com/questions/5349867/error-handling-in-asp-net-mvc3
Update: if it doesn't help - read this discussion (especially the last post on page 2): http://forums.asp.net/t/1657582.aspx/2/10?Error+500+0+when+MVC+runs+on+IIS7
Update 2: This one looks very promissing too: http://forums.asp.net/t/1652026.aspx/1
remove <add verb="POST,GET" path="ajax/*.ashx" type="Ajax.PageHandlerFactory, Ajax" /> from httpHandlers
and add <add verb="POST,GET" name="scripthandlerajaxFactory" path="ajax/*.ashx" type="Ajax.PageHandlerFactory, Ajax" /> to handlers