I found that when using curl http://localhost:8080 it works, but when using node-fetch like the webid-provider-tests test suite does, you get a connection refused error.
Running

HOST=127.0.0.1 composer serve-dev

Fixes this! :/
I'll see if I can make node-fetch send the same Host header as curl, so we can stop having to do this.