0

I have a MySQL database server running on a VPS. The server is connected to a web server via a private network interface, and has no public services except SSH and a PHPMyAdmin installation.

Some time ago, I installed a Monit rule to check for abnormal traffic over the Public network interface (AKA, not the one that MySQL data goes over):

# Monitor network connection check network public with interface eth1 if failed link then alert if changed link then alert if saturation > 90% then alert if download > 10 MB/s then alert if total upload > 1 GB in last hour then alert

A few months ago, I started getting flurries of Monit alerts triggered by this rule:

Upload bytes exceeded Service public Date: Tue, 04 Apr 2017 13:11:55 Action: alert Host: myhostname.com Description: total upload 1.0 GB matches limit [upload rate < 1024.0 MB in last 1 hour] Your faithful employee, Monit

These alerts don't really correlate with any spike of usage on the server, and since there are no real public-facing services I can't imagine what could be responsible for this amount of upload bandwidth. Disk space doesn't seem to be being used either.

How can I find out what is causing 1GB+ of "something" to be uploaded, what is being uploaded, and from where?

Improve this question
3
  • Do you control the server ? Commented Apr 4, 2017 at 18:53
  • If MySQL should only be used over the private interface, you could block port 3306 on the public interface in your firewall rules. Commented Apr 4, 2017 at 20:55
  • I control the server, and 3306 (as well as most other ports) are blocked on the public interface. Commented Apr 5, 2017 at 2:16

2 Answers 2

Reset to default
1

Sounds like something is running some kind of periodic SELECT query that's dumping data out the public interface.

Wireshark, pcap, or other variant of packet snoop would reveal source (the SQL server) and target, ports/protocols used, and so forth. Zero in on port 3306 (either TCP or UDP), which is the MySQL port.

If you've a lot of disk space on the system you can set your packet snoop up to save the pcap files containing the network traffic over your public interface.

I would recommend spooling mode, say, 10GB, broken into 2GB files, such that you never use more that that amount of space, but you should be able to capture at least one of these uploads (or at least the start of it). It may take an hour or two to fill that, or only a few minutes, but you'll always have at least 8GB of intact conversation using this method (the last file will be <2GB and will be the newest data).

Once you have an idea of what's happening, you can take steps to mitigate. Right off the bat, I'd recommend closing port 3306 on the public port and also locking down "May run queries" creds within MySQL's permissions. That'll lock the barn door.

Improve this answer
0

A possible a way to identify what is happening from the MySQL side is to enable the slow query log then establish a really low threshold to capture every statement. Then you can look through the slow query log to see what was running at that time.

An alternative approach would be to dump "show full processlist \G" periodically. This will be much easier to see what is running at a given time.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.