2

I have a remote Ubuntu system hooked up to a mobile network. The remote system is running a couple of services, where it is uploading images from a camera to a database (Google Firebase). For a while now, I have had the suspicion that the remote system consumes more network ressources than corresponding to the size of the images it uploads. How can I measure network usage across the system and its processes?

System specifics:

  • Jetson Orin Nx
  • Nvidia Jetpack 6.0 (linux 4 tegra version 36.3)
  • Includes linux kernel 5.15 and based off Ubuntu 22.04
  • Internet access via a Teltonika TRB140 4G-gateway
  • configured with OpenVPN for remote access through Access Server

Here is how far I am currently:

I have installed netstat and have used it to monitor traffic on two network devices eth0 & tun0 for a few weeks now. OpenVPN is set up to redirect all traffif to IPv4 addresses through the tunnel, except traffic to the OpenVPN server itself. So as expected most days tun0 sees a little less traffic (e.g. rx for eth0 would be 124.27 MiB and for tun0 would be 115.67 MiB). However I have also seen days where there has been a large difference in traffic between the two devices (e.g. rx for eth0 could be 1.17 GiB and for tun0 169.57 MiB). So far I have not been able to figure out what causes such a big discrepancy.

I have used tcpdump to monitor the traffic at eth0.

15:16:27.120298 IP [myhost.local].39090 > [myrouter.local].domain: 43310+ [1au] AAAA? connectivity-check.ubuntu.com. (58) 15:16:27.127602 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.144210 IP [myrouter.local].domain > [myhost.local].39090: 43310 12/0/1 AAAA 2620:2d:4000:1::97, AAAA 2620:2d:4000:1::2b, AAAA 2620:2d:4002:1::198, AAAA 2001:67c:1562::24, AAAA 2620:2d:4000:1::98, AAAA 2620:2d:4000:1::96, AAAA 2001:67c:1562::23, AAAA 2620:2d:4002:1::197, AAAA 2620:2d:4000:1::23, AAAA 2620:2d:4002:1::196, AAAA 2620:2d:4000:1::2a, AAAA 2620:2d:4000:1::22 (394) 15:16:27.146016 IP6 [myhost].49180 > ubuntu-content-cache-2.ps5.canonical.com.http: Flags [S], seq 1128295973, win 64800, options [mss 1440,sackOK,TS val 3360569444 ecr 0,nop,wscale 7], length 0 15:16:27.161670 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 556 15:16:27.176553 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.182545 IP6 ubuntu-content-cache-2.ps5.canonical.com.http > [myhost].49180: Flags [S.], seq 3562386702, ack 1128295974, win 64260, options [mss 1396,sackOK,TS val 424163785 ecr 3360569444,nop,wscale 14], length 0 15:16:27.182648 IP6 [myhost].49180 > ubuntu-content-cache-2.ps5.canonical.com.http: Flags [.], ack 1, win 507, options [nop,nop,TS val 3360569481 ecr 424163785], length 0 15:16:27.182882 IP6 [myhost].49180 > ubuntu-content-cache-2.ps5.canonical.com.http: Flags [P.], seq 1:88, ack 1, win 507, options [nop,nop,TS val 3360569481 ecr 424163785], length 87: HTTP: GET / HTTP/1.1 15:16:27.221864 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 772 15:16:27.223004 IP [myhost.local].37861 > [myrouter.local].domain: 54341+ [1au] PTR? 7.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.0.0.0.4.d.2.0.0.0.2.6.2.ip6.arpa. (101) 15:16:27.224824 IP6 ubuntu-content-cache-2.ps5.canonical.com.http > [myhost].49180: Flags [P.], seq 1:186, ack 88, win 4, options [nop,nop,TS val 424163828 ecr 3360569481], length 185: HTTP: HTTP/1.1 204 No Content 15:16:27.224824 IP6 ubuntu-content-cache-2.ps5.canonical.com.http > [myhost].49180: Flags [F.], seq 186, ack 88, win 4, options [nop,nop,TS val 424163828 ecr 3360569481], length 0 15:16:27.224924 IP6 [myhost].49180 > ubuntu-content-cache-2.ps5.canonical.com.http: Flags [.], ack 186, win 506, options [nop,nop,TS val 3360569523 ecr 424163828], length 0 15:16:27.225162 IP6 [myhost].49180 > ubuntu-content-cache-2.ps5.canonical.com.http: Flags [F.], seq 88, ack 187, win 506, options [nop,nop,TS val 3360569524 ecr 424163828], length 0 15:16:27.233551 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.265856 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 556 15:16:27.267544 IP6 ubuntu-content-cache-2.ps5.canonical.com.http > [myhost].49180: Flags [.], ack 89, win 4, options [nop,nop,TS val 424163871 ecr 3360569524], length 0 15:16:27.280539 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.293154 IP [myrouter.local].domain > [myhost.local].37861: 54341 1/0/1 PTR ubuntu-content-cache-2.ps5.canonical.com. (155) 15:16:27.294947 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 468 15:16:27.294990 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 716 15:16:27.329822 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 756 15:16:27.329886 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 636 15:16:27.329937 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 652 15:16:27.339681 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.365533 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.369906 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 1156 15:16:27.388740 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.400514 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.438336 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 772 15:16:27.450757 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.473690 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 660 15:16:27.483692 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.541877 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 436 15:16:27.571529 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.577683 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 556 15:16:27.619711 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.645845 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 436 15:16:27.681894 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 452 15:16:27.689529 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.697515 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.749868 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 436 15:16:27.768818 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.785988 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 660 15:16:27.813550 IP [myhost.local].51534 > [myrouter.local].domain: 19198+ [1au] SRV? _grpclb._tcp.firestore.googleapis.com. (66) 15:16:27.814004 IP [myrouter.local].domain > [myhost.local].51534: 19198 NXDomain 0/0/1 (66) 15:16:27.814158 IP [myhost.local].51534 > [myrouter.local].domain: 19198+ SRV? _grpclb._tcp.firestore.googleapis.com. (55) 15:16:27.814509 IP [myrouter.local].domain > [myhost.local].51534: 19198 NXDomain 0/0/0 (55) 15:16:27.820606 IP [VPN_SERVER_IP].openvpn > [myhost.local].33349: UDP, length 64 15:16:27.853922 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 476 15:16:27.853994 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 308 15:16:27.854035 IP [myhost.local].33349 > [VPN_SERVER_IP].openvpn: UDP, length 276 15:16:27.855088 IP6 _gateway > [myhost]: ICMP6, neighbor solicitation, who has [myhost], length 32 15:16:27.855168 IP6 [myhost] > _gateway: ICMP6, neighbor advertisement, tgt is [myhost], length 24

As far as I can see, there are some DNS queries that go directly out of eth0 as well as some http requests to Canonical for a captive-portal probe over IPv6. But so far have not yet seen anythin that can explain the big discrepancy.

Now comparing the data from vnstat and the billing from the network provider, I have observed a pattern where for the past four weeks, every Saturday there has been more than 1 GiB of traffic on the rx side, where I would expect somewhere in the range of 5-20 MiB. But it can also occur other days. I have set up two cronjobs to export the hourly data from vnstat from both devices every day and I can see from the two most recent data spikes that the traffic occurs between 6 AM and 7 AM: (Following is rx | tx | total | avg. rate)

  • Saturday 2025-07-05 had 1.09 GiB | 44.15 MiB | 1.13 GiB | 2.70 Mbit/s between 6 AM and 7 AM
  • Tuesday 2025-07-08 had 945.94 MiB | 39.91 MiB | 985.85 MiB | 2.30 Mbit/s also between 6 AM and 7 AM

I have also set up IP-accounting for the three services, that I have deployed and from systemctl status [myservice.service] only one seems to be using network ressources. Here is the sanitized output from that service:

myservice.service - Custom Python Service Loaded: loaded (/etc/systemd/system/example-myservice.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system.control/myservice.service.d └─50-IPAccounting.conf Active: active (running) since Thu 1970-01-01 01:00:45 CET; 55 years 6 months ago Main PID: 2051 (pt_main_thread) IP: 204.4M in, 472.4M out Tasks: 261 (limit: 8814) Memory: 2.0G CPU: 5d 20h 50min 13.318s CGroup: /system.slice/myservice.service ├─2051 /usr/bin/python3 myproject/scripts/main.py ├─3581 /home/username/.local/bin/geckodriver --port 49249 --websocket-port 47211 ├─3584 /usr/lib/firefox/firefox --marionette --kiosk --remote-debugging-port 47211 ... ├─3661 /usr/lib/firefox/firefox -contentproc ... └─[... more firefox subprocesses omitted ...]

I set up IP-accounting for the services on 2025-06-30 so and with only 204.4M received for this service in that time frame it does not seem like this service is the cause of the spike in network traffic.

So what tool can I use for monitoring the network traffic across the services I have deployed and other processes that could be causing the data spikes? Knowing that it seems data spikes happen between 6 AM and 7 AM on some days during the week and likely not from processes I have deployed myself, is there a tool I can use to log data traffic in this timeframe and saving the output for later reading?

Improve this question
8
  • 2
    > Does all tun0 traffic also go through eth0 or is it seperate traffic going through the two devices? We can't tell you how you've set up your VPN! Commented Jul 1 at 10:12
  • 1
    Fair point, I took over this project from someone else, so right now don't know the specifics of how OpenVPN has been configured. But is on my list of todos, I only added the information and the bonus question in the hope it added additional useful context Commented Jul 1 at 11:01
  • I guess the VPN packets on tun0 are encapsulated and sent over the real network interface eth0. This could be an explanation for 124.27 MiB vs. 115.67 MiB. If the additional traffic on eth0 is TCP, then you can try running tcpdump and capture SYN and FIN packets using something like tcpdump 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0' with writing to a file and later check the IP addresses and port numbers. Maybe you can guess what could be causing the additional traffic. Commented Jul 1 at 14:24
  • 1
    If you're running tcpdump over, e.g., an ssh connection on the same interface you're dumping, you will want to exclude that ssh connection from the tcpdump (e.g. add and not (port ssh and host x.x.x.x)` to the tcpdump expression, where x.x.x.x is your IP address or hostname). Otherwise every packet monitored will generate more traffic over the ssh connection, which will also be monitored, resulting in even more traffic, and so on. The interesting stuff you're looking for will be buried in the noise of all the ssh traffic. Commented Jul 2 at 6:42
  • 1
    @christian-37 You should edit your question and add all information to the question. It is difficult to get the whole picture by reading all comments. Instead of only describing some commands output with your own words, also show the exact output in the question. This helps us to verify that your interpretation is correct. Commented Jul 3 at 10:50

1 Answer 1

Reset to default
2

There's a tool named jnettop (at least on Debian, should be available on Ubuntu as well), that measures current traffic over individual sockets. That is, it shows a list of localIP:localPort <--> remoteIP:remotePort pairs and tells you the current bandwidth on each of them.

The problem is, as the name suggests, it's a top-like tool - shows you what's going on at the moment, but at least to my knowledge does not aggregate data, like vnstat does.

Still, you may find it useful if you happen to catch a period of high traffic. At the very least you can confirm whether tun0 traffic is included in eth0 traffic or not. (It can filter by individual interfaces or devices.)

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.