- Notifications
You must be signed in to change notification settings - Fork 102
feat: introducing bulk read API through Batcher #99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
...rc/main/java/com/google/cloud/bigtable/data/v2/stub/readrows/ReadRowsBatchingDescriptor.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
...loud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStub.java Outdated Show resolved Hide resolved
...rc/main/java/com/google/cloud/bigtable/data/v2/stub/readrows/ReadRowsBatchingDescriptor.java Outdated Show resolved Hide resolved
...rc/main/java/com/google/cloud/bigtable/data/v2/stub/readrows/ReadRowsBatchingDescriptor.java Outdated Show resolved Hide resolved
Codecov Report
@@ Coverage Diff @@ ## master #99 +/- ## ========================================= Coverage ? 81.92% Complexity ? 971 ========================================= Files ? 99 Lines ? 6021 Branches ? 330 ========================================= Hits ? 4933 Misses ? 910 Partials ? 178
Continue to review full report at Codecov.
|
This change introduces BulkReadAPI on BigtableDataClient. This operation accepts row keys in a batch mode and behind the scene fetch rows based on configurable batches.
google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/BigtableDataClient.java Show resolved Hide resolved
google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/BigtableDataClient.java Show resolved Hide resolved
google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/models/Query.java Outdated Show resolved Hide resolved
...loud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStub.java Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/BigtableDataClient.java Outdated Show resolved Hide resolved
google-cloud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/BigtableDataClient.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
| .setTableName(NameUtil.formatTableName(PROJECT_ID, INSTANCE_ID, TABLE_ID)) | ||
| .setAppProfileId(APP_PROFILE_ID) | ||
| .setRowsLimit(10) | ||
| .build()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are you testing here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for pointing it out, Have updated this with valid test case.
google-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/it/BulkReadIT.java Outdated Show resolved Hide resolved
| /** Returns a builder for the default ChannelProvider for this service. */ | ||
| public static InstantiatingGrpcChannelProvider.Builder defaultGrpcTransportProviderBuilder() { | ||
| return BigtableStubSettings.defaultGrpcTransportProviderBuilder() | ||
| // TODO: tune channels |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
todo should move to getDefaultChannelPoolSize
igorbernstein2 left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@kolea2 do you want to take a pass as well?
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
| BatchingSettings.newBuilder() | ||
| .setIsEnabled(true) | ||
| .setElementCountThreshold(maxElementPerBatch) | ||
| .setRequestByteThreshold(400L * 1024L) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we make this a variable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could have a variable here. Although as in this format, it is similar to the existing bulkMutateRowsSettings configuration
Lines 500 to 516 in abbc6c8
| bulkMutateRowsSettings = | |
| BigtableBatchingCallSettings.newBuilder(new MutateRowsBatchingDescriptor()) | |
| .setRetryableCodes(IDEMPOTENT_RETRY_CODES) | |
| .setRetrySettings(MUTATE_ROWS_RETRY_SETTINGS) | |
| .setBatchingSettings( | |
| BatchingSettings.newBuilder() | |
| .setIsEnabled(true) | |
| .setElementCountThreshold(100L) | |
| .setRequestByteThreshold(20L * 1024 * 1024) | |
| .setDelayThreshold(Duration.ofSeconds(1)) | |
| .setFlowControlSettings( | |
| FlowControlSettings.newBuilder() | |
| .setLimitExceededBehavior(LimitExceededBehavior.Block) | |
| .setMaxOutstandingRequestBytes(100L * 1024 * 1024) | |
| .setMaxOutstandingElementCount(1_000L) | |
| .build()) | |
| .build()); |
We might need to change that to have common formatting in separate PR
...table/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStubSettings.java Outdated Show resolved Hide resolved
google-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/it/BulkReadIT.java Outdated Show resolved Hide resolved
| Added a few small comments but LGTM overall! |
Fixes #7
This change introduces BulkReadAPI on BigtableDataClient. This operation
accepts row keys in a batch mode and behind the scene split them based
on configurable values.