Regarding the latency issue when using a relay with two nodes

[ISensuiI]

Regarding the latency issue when using a relay with two nodes:

I'm currently developing a product using iRoh to provide online connectivity for a game.

Node A: My physical machine. Latency with Relay A: 70ms.

Node B: Virtual machine (VM). Latency with the relay: 70ms. Latency with Node A: 1ms. They are on a local area network.

My current testing environment involves two nodes. Endpoint builder section:

The relay URL and pkarr URL are custom services I set; I'm not using the official iRoH services. I'm sure they work correctly. let endpoint = Endpoint::empty_builder(RelayMode::Custom(relay_manager.get_relay_map())) .path_selection(PathSelection::RelayOnly) .secret_key(secret_key) .discovery(PkarrPublisher::builder(pkarr_manager.pkarr_url.clone())) .discovery(PkarrResolver::builder( pkarr_manager.pkarr_url.clone())) .discovery(DnsDiscovery::builder(config.iroh.dns_origin.clone())) .bind() .await .context(EndpointBuildFailedSnafu)?;` 

They remove .path_selection(PathSelection::RelayOnly) After a successful hole punch, the latency is approximately 3-8ms. The extra 5ms is likely due to iRoH's own latency, which is understandable.

However, when simulating a scenario where two nodes in a symmetrical network, or one node in a symmetrical network, cannot successfully punch a hole and must communicate via a relay, the latency jumps to 270-400ms (using CMD PING X.X.X.X -t).

I understand that a 70ms latency for the relay server is already relatively high, and we can upgrade to a lower 30-40ms server for our final production environment.

The problem is that I don't understand how the 400ms latency is generated. Is it a configuration issue?

My tentative guess is that the overall 400ms latency is caused by:
NodeA -> Relay (Ping 70ms)
Relay -> NodeB (Ping 70ms)
NodeB -> Relay (Pong 70ms)
Relay -> NodeA (Pong 70ms)

70ms * 4 ≈ 280-400ms

That's too high a latency for a game. I think a normal latency, like my server's 30-40ms, or even 100ms for two nodes with a relay, is reasonable. But this is too high. I don't know if this is typical of iRoH. Should I try to fix it with my relay server configuration, or is there something wrong with my setup?

Below are samples:

Without using iroh, NODEA PING NODE B from 10.0.0.10: Bytes=32 Time=1ms TTL=128 from 10.0.0.10: Bytes=32 Time=1ms TTL=128 from 10.0.0.10: Bytes=32 Time=1ms TTL=128 Using iroh and successfully punching a hole, NODEA PING NODE B from 10.0.0.10: Bytes=32 Time=5ms TTL=128 from 10.0.0.10: Bytes=32 Time=3ms TTL=128 from 10.0.0.10: Bytes=32 Time=5ms TTL=128 from 10.0.0.10: Bytes=32 Time=4ms TTL=128 from 10.0.0.10: Bytes=32 Time=4ms TTL=128 from 10.0.0.10: Bytes=32 Time=5ms TTL=128 Using iroh but PathSelection::RelayOnly NODEA PING NODE B from 10.0.0.10: Bytes=32 Time=260ms TTL=128 from 10.0.0.10: Bytes=32 Time=337ms TTL=128 from 10.0.0.10: Bytes=32 Time=341ms TTL=128 from 10.0.0.10: Bytes=32 Time=250ms TTL=128 From 10.0.0.10: Bytes=32 Time=425ms TTL=128 

[matheus23]

My tentative guess is that the overall 400ms latency is caused by:
NodeA -> Relay (Ping 70ms)
Relay -> NodeB (Ping 70ms)
NodeB -> Relay (Pong 70ms)
Relay -> NodeA (Pong 70ms)

The arrows are correct, this is what needs to happen.

The latency however confuses me. "Latency" can mean two things: round-trip-time or single-trip-time.

In this case it looks like you're describing single-trip-time.

Usually we measure latency in round-trip-time. If you get 70ms reported as latency in iroh, then that should mean round-trip-time.

In that case, the message should only take 35ms from NodeA to Relay, and so on. So in total it should only take 140ms with 70ms of round-trip-time to the relay for both nodes (assuming they're connected to the same relay).

If you can manage to deploy an iroh relay server with, let's say, 30ms round-trip-time to clients, then the RTT to another client should be 60ms approximately.

[matheus23]

I actually just ping the relay server's IP address (150.1.1.1) directly

Ah okay!

But as soon as onlyrelay is enabled, it becomes 300ms-400ms. I can understand the higher latency when using onlyrelay, but I don't understand why it's 300ms-400ms.

Yeah that's indeed confusing.

One thing to consider is that usually latency increases with throughput. So if you're sending close to the maximum amount of data you can send on the relay path, then you'll experience higher latency.

That said, it looks like all you're sending is just small pings, so those should not increase latency measurably.

---

Let's continue the discussion in the issue #3699

[ISensuiI]

Yes, that's what puzzled me too, which is why I didn't initially raise an issue. I'm not sure if it was a misunderstanding on my part.

Now that you understand my problem, let me add some more information. Regarding my relay server, it's an 8GB 4-core server with 10Mbps bandwidth, although I think these are irrelevant.

Because I've used libp2p extensively before, the relay acted as a transparent layer, simply helping both sides find connections. Even after switching to onlyrelay mode, I believe the relay remains a transparent layer, only now it helps forward connections between nodes, and as long as my server's bandwidth is sufficient, it shouldn't impose any restrictions. Furthermore, I've kept all default configurations in iroh-relay and haven't found any related limitations. Even if there were, I believe our current setup is just a simple ping with a very small data volume, so there shouldn't be any latency. And even with large amounts of data, a properly configured iroh-relay should be able to handle the load.

Based on current performance tests, it's insufficient to support my current and future business. I still can't figure out where the problem lies.

I had another idea before, which was to use the official server for comparative testing. The problem is that I'm almost certain that the official iRoh relay server wouldn't be using onlyRelay. So I eventually abandoned that idea.

This is some code related to relay setup. about iroh-dns and pkarr, I believe they are irrelevant to the current issue, so I haven't pasted them here.

 pub async fn new(params: RelayServerParams) -> AppResult<Self> { info!(target: "app::relay::build", "RelayServer initializing with clean parameters..."); let cert_loader = MyCertLoader { cert_path: params.cert_pem_path, key_path: params.cert_key_path, }; let reloading_resolver = ReloadingResolver::init( cert_loader, Duration::from_secs(12 * 60 * 60), ).await?; let tls_server_config = rustls::ServerConfig::builder_with_provider( Arc::new(rustls::crypto::ring::default_provider()), ) .with_safe_default_protocol_versions() .expect("协议错误") .with_no_client_auth() .with_cert_resolver(Arc::new(reloading_resolver)); let cert_config = CertConfig::<Box<dyn Debug>, Box<dyn Debug>>::Reloading; let tls_config = TlsConfig { https_bind_addr: SocketAddr::new("0.0.0.0".parse().unwrap(), params.https_port), quic_bind_addr: SocketAddr::new("0.0.0.0".parse().unwrap(), params.quic_port), cert: cert_config, server_config: tls_server_config.clone(), }; let relay_config = RelayConfig { http_bind_addr: SocketAddr::new("0.0.0.0".parse().unwrap(), params.http_port), tls: Some(tls_config), limits: Default::default(), key_cache_capacity: Some(32), access: AccessConfig::Everyone, }; let quic_config = QuicConfig { bind_addr: SocketAddr::new(IpAddr::V4(Ipv4Addr::new(0, 0, 0, 0)), params.quic_port), server_config: tls_server_config }; let relay_config = ServerConfig { relay: Some(relay_config), quic: Some(quic_config), metrics_addr: Some(SocketAddr::new(IpAddr::V4(Ipv4Addr::new(0, 0, 0, 0)), DEFAULT_METRICS_PORT)), }; let relay_server = Server::spawn(relay_config).await.map_err(|e| { let error_message = e.to_string(); AppError::System { message: format!("Relay 服务启动失败: {}", error_message), source: None, } })?; info!(target: "app::relay::build", "RelayServer initialized successfully."); Ok(Self { server: Some(relay_server), }) } 

[matheus23]

I had another idea before, which was to use the official server for comparative testing. The problem is that I'm almost certain that the official iRoh relay server wouldn't be using onlyRelay. So I eventually abandoned that idea.

PathSelection::RelayOnly is purely an Endpoint-specific configuration value. The relay server can't tell if clients enabled this setting or not.

[ISensuiI]

Oh dear, I was completely confused by this problem. You reminded me; I had mixed it up earlier. OnlyRelay is the client-side component. I'll change all the service addresses to the official iRoh addresses and then test and compare them. That way, I'll be able to pinpoint the specific issue.

[ISensuiI]

I changed the configuration from custom DNS relay to the official configuration. The code is as follows:

let endpoint = Endpoint::builder() .path_selection(PathSelection::RelayOnly) .secret_key(secret_key) .bind() .await .context(EndpointBuildFailedSnafu)?; 

The test results are as follows:

Ping the official server directly

Ping usw1-1.relay.n0.iroh-canary.iroh.link [5.78.107.194] with 32 bytes of data:

From 5.78.107.194 Reply from 5.78.107.194: Bytes=32 Time=238ms TTL=41

Reply from 5.78.107.194: Bytes=32 Time=238ms TTL=41

Reply from 5.78.107.194: Bytes=32 Time=238ms TTL=41

Reply from 5.78.107.194: Bytes=32 Time=238ms TTL=41

Without RelayOnly enabled

iroh+tun test

Reply from 10.0.0.10: Bytes=32 Time=1ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=1ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=1ms TTL=128

From Reply from 10.0.0.10: Bytes=32 Time=12ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=4ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=5ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=5ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=5ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=3ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=3ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=4ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=5ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=5ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=3ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=3ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=4ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=4ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=3ms TTL=128

Enable RelayOnly

From Reply from 10.0.0.10: Bytes=32 Time=760ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=725ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=927ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=766ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=907ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=779ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=505ms TTL=128

From Reply from 10.0.0.10: Bytes=32 Time=936ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=971ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=947ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=696ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=1048ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=535ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=727ms TTL=128

From Reply from 10.0.0.10: Bytes=32 Time=1028ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=639ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=797ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=722ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=962ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=913ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=891ms TTL=128

From Reply from 10.0.0.10: Bytes=32 Time=875ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=625ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=911ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=874ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=931ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=1154ms TTL=128

Reply from 10.0.0.10: Bytes=32 Time=837ms TTL=128

---

If we follow my previous algorithm:

usw1-1.relay.n0.iroh-canary.iroh.link [5.78.107.194] Ping 240ms

Owd ≈ 120ms

120 * 4 ≈ 480 Theoretically, under stable network conditions, the fluctuation should not exceed 20-30ms, meaning it should fluctuate around 480-510ms. However, the above results average around 700ms, and this is only for ping scenarios; if data transmission latency is involved, it will be even higher.

This comparison proves that it's not a problem with my custom relay or iroh-dns.

These are some relevant logs that prove the specific service address I connected to.

{"timestamp":"2025-11-25 19:55:07.076","level":"DEBUG","fields":{"message":"Pong received from relay server","data":"[87, 95, 119, 203, 221, 71, 44, 46]"},"target":"iroh_relay::ping_tracker","span":{"name":"connected"},"spans":[{"name":"new"},{"name":"relay-actor"},{"url":"https://usw1-1.relay.n0.iroh-canary.iroh.link./","name":"active-relay"},{"name":"connected"}]} {"timestamp":"2025-11-25 19:55:07.448","level":"DEBUG","fields":{"message":"Sending ping to relay server.","data":"[183, 218, 3, 62, 135, 238, 246, 109]"},"target":"iroh_relay::ping_tracker","span":{"name":"connected"},"spans":[{"name":"new"},{"name":"relay-actor"},{"url":"https://usw1-1.relay.n0.iroh-canary.iroh.link./","name":"active-relay"},{"name":"connected"}]} {"timestamp":"2025-11-25 19:55:05.267","level":"DEBUG","fields":{"message":"Publish endpoint info to pkarr","data":"EndpointData { addrs: {Ip(10.0.0.100:58378), Ip(192.168.1.173:58378), Ip(192.168.3.1:58378), Ip(192.168.88.1:58378)}, user_data: None }","pkarr_relay":"https://dns.iroh.link/pkarr"},"target":"iroh::discovery::pkarr","span":{"me":"69104ef20c","name":"pkarr_publish"},"spans":[{"name":"new"},{"me":"69104ef20c","name":"pkarr_publish"}]}