I had quite some trouble getting a custom 400 error document to work. I added this to the top level of my Apache conf:
ErrorDocument 404 /404.html ErrorDocument 400 /400.html
And created the two error documents in the DocumentRoot. First, I tested the 404 with:
echo -en "GET /foo HTTP/1.0\n\n" | nc 127.0.0.1 80
This returned my custom 404.
Then the 400 (Note that the leading slash is missing):
echo -en "GET foo HTTP/1.0\n\n" | nc 127.0.0.1 80
The response was the default Apache 400.
I did manage to get my custom 400 back by talking HTTP to an HTTPS vhost:
echo -en "GET / HTTP/1.0\n\n" | nc 127.0.0.1 443
You still get the same custom 400 back, even if you make an invalid request:
echo -en "GET foo HTTP/1.0\n\n" | nc 127.0.0.1 443
Unfortunately for your testing, talking SSL to a non-ssl vhost doesn't result in a 400 of any sort. I still haven't figured out how to get a custom ErrorDocuemnt 400 to be sent back on port 80.
P.S. I remember reading somewhere that an ErrorDocument 400 won't work inside a .htaccess file as it may not be possible to even determine the correct DocumentRoot if the request is malformed. The same probably applies to name based virtualhosts.
ErrorDocument 400for some requests.