2

We have a Bitbucket Server instance hosted in AWS. From some other AWS servers (in another region), git clone over SSH fails with

ssh: connect to host (hostname) port 7999: Connection refused

However, other servers in AWS (in the same region as the Bitbucket server) can successfully clone over SSH, using the same URL.

Other information:

  • Bitbucket is definitely listening on port 7999:

    $ sudo netstat -tnlp | grep :7999 tcp6 0 0 :::7999 :::* LISTEN 20707/java

    (process 20707 is the main Bitbucket process)

  • Bitbucket is running behind Apache as a reverse proxy to provide SSL.

  • tcptraceroute and mtr on port 7999 from the instance that can't clone to Bitbucket on port 7999 successfully connect.
  • The public key for the keypair used by the instance that can't connect is added to the repository's access keys with read access on Bitbucket.
  • In the AWS security groups, the cloning instance's group allows all outbound connections, and the Bitbucket server's groups allow connections from anywhere on port 7999.

  • Results from a tcpdump on port 7999 on the cloning instance:

    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 16:56:53.348387 IP (tos 0x0, ttl 64, id 61715, offset 0, flags [DF], proto TCP (6), length 60) (cloning server's hostname).38606 > (bitbucket server's hostname).irdmi2: Flags [S], cksum 0x2799 (incorrect -> 0xb1db), seq 3675985409, win 26883, options [mss 8961,sackOK,TS val 1512892178 ecr 0,nop,wscale 7], length 0 16:56:53.489908 IP (tos 0x0, ttl 252, id 37586, offset 0, flags [none], proto TCP (6), length 40) (bitbucket server's hostname).irdmi2 > (cloning server's hostname).38606: Flags [R.], cksum 0x24e4 (correct), seq 1472002966, ack 3675985410, win 26883, length 0

  • Results from ssh'ing from the cloning server to Bitbucket:

     $ sudo ssh -vvv -p 7999 ssh://[email protected] OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /root/.ssh/config debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 56: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to stash.tddevops.com [172.24.16.201] port 7999. debug1: connect to address 172.24.16.201 port 7999: Connection refused ssh: connect to host stash.tddevops.com port 7999: Connection refused

  • tcpdump on the Bitbucket server shows no data when a clone is attempted.

Improve this question
13
  • On the 'other AWS server (in another region)', run tcpdump -n port 7999 and compare traffic caused by git clone ... and telnet <bitbucketserver> 7999 (assuming telnet succeeds, if not use tcptraceroute). Commented Oct 31, 2017 at 16:27
  • @sborsky There's a lot more traffic caused by git clone, as one might expect, but I'm not sure what differences I'm looking for. Commented Oct 31, 2017 at 16:54
  • You're looking for the TCP request sent by git, that results in Connection refused. Compare that to a successful one (e.g. sent by telnet). The difference should give a clue on what is going wrong. Commented Oct 31, 2017 at 16:58
  • @sborsky I added the tcpdump output from the cloning instance to the question; does that help? Commented Oct 31, 2017 at 17:09
  • Could you post the tcpdump -n output for a successful connection, to compare? Commented Oct 31, 2017 at 18:48

2 Answers 2

Reset to default
0

From https://serverfault.com/a/288493/442063:

You see the "incorrect" checksums due to a feature called TCP checksum offloading. The checksum fields for outgoing TCP packets are not pre-calculated by the operating system but instead set to 0 and left for calculation by the NIC processor. The Wireshark FAQ has a more detailed explanation.

That might help, but your problem sounds like a firewall issue. It couldn't hurt to make sure you do not have an internal firewall running on this hosted AWS server. Without knowing what OS you are running, I can't really speculate on what to look for. Many Linux operating systems use iptables. For example, you can see here how to turn off and disable the firewall for Oracle Linux or Redhat Linux.

Improve this answer
1
  • It's a CentOS server, for the record. I checked with iptables -L; there aren't any rules, iptables allows any connections. Commented Nov 6, 2017 at 15:35
0

I found out the issue; it was an external issue with our Security department's firewall. Nothing needed to be changed on the cloning server or the Bitbucket server.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.