Skip to content

Commit 25ae79a

Browse files
mina-ashamMina Asham
authored andcommitted
fix(bigquery): Make exponential backoff retry second based
- Currently these retries are doing retries in milliseconds, which is way too aggressive, the current retry steps are (1ms, 2ms, 4ms, 8ms, 16ms, 32ms, 64ms, 128ms, 256ms, 512ms, 1024ms, ...), it's highly likely the first 7 retries will fail with rate limiting specially in a big workload (specifically this is causing an issue in the Spark bigquery connector), I suspect the exponential retries here were meant to be in seconds (though even if it wasn't meant to, this might be a slightly better approach to reduce the load on the servers - The new steps would be: 1s, 2s, 4s, 8s, 16s, 32s, 60s (repeated till we exhaust the 5 minutes max retries)
1 parent 7d935f0 commit 25ae79a

File tree

2 files changed

+3
-3
lines changed

2 files changed

+3
-3
lines changed

google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorker.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -502,7 +502,7 @@ private void maybeWaitForInflightQuota() {
502502

503503
@VisibleForTesting
504504
static long calculateSleepTimeMilli(long retryCount) {
505-
return Math.min((long) Math.pow(2, retryCount), 60000);
505+
return Math.min((long) Math.pow(2, retryCount) * 1000, 60000);
506506
}
507507

508508
@VisibleForTesting

google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/ConnectionWorkerTest.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -502,8 +502,8 @@ public void testStreamNameMismatch() throws Exception {
502502

503503
@Test
504504
public void testExponentialBackoff() throws Exception {
505-
assertThat(ConnectionWorker.calculateSleepTimeMilli(0)).isEqualTo(1);
506-
assertThat(ConnectionWorker.calculateSleepTimeMilli(5)).isEqualTo(32);
505+
assertThat(ConnectionWorker.calculateSleepTimeMilli(0)).isEqualTo(1000);
506+
assertThat(ConnectionWorker.calculateSleepTimeMilli(5)).isEqualTo(32000);
507507
assertThat(ConnectionWorker.calculateSleepTimeMilli(100)).isEqualTo(60000);
508508
}
509509

0 commit comments

Comments
 (0)