Using WordPress Jetpack with Nginx and Varnish

I need to manage a few WordPress sites and I wanted to add the Jetpack plugin so I can manage all sites from a WordPress.com account. After having installed the Jetpack plugin, trying to manage my site from WordPress.com fails with this error:
Screen Shot 2015-07-03 at 23.25.44

Watching the Varnish log, I tried the same call made by WordPress.com with curl. The endpoint /xmlrpc.php is called via POST, and curl gave me another hint:

<?xml version="1.0" encoding="UTF-8"?>
<methodResponse>
 <fault>
 <value>
 <struct>
 <member>
 <name>faultCode</name>
 <value><int>-32700</int></value>
 </member>
 <member>
 <name>faultString</name>
 <value><string>parse error. not well formed</string></value>
 </member>
 </struct>
 </value>
 </fault>
</methodResponse>

The error -32700 is not really useful, but a search on DuckDuckGo gave me the solution:

Add this line to the beginning of the wp_config.php file of your site:

$_SERVER['SERVER_PORT'] = 80;

And voila, it works.

Almost everybody seems to agree on Varnish being the problem, but the solutions given did not work for me. Just in case, the solutions mentioned were to tell Varnish to ignore the calls to /xmlrpc.php.