- Notifications
You must be signed in to change notification settings - Fork 35
feat: update cloudstorageconfiguration to improve copy accross cross-bucket performance #168
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
@@ Coverage Diff @@ ## master #168 +/- ## ============================================ + Coverage 71.13% 71.79% +0.66% - Complexity 481 495 +14 ============================================ Files 29 29 Lines 1642 1656 +14 Branches 265 268 +3 ============================================ + Hits 1168 1189 +21 + Misses 348 342 -6 + Partials 126 125 -1
Continue to review full report at Codecov.
|
| assertNotEquals(sourceFileSystem.config(), destFileSystem.config()); | ||
| assertEquals("bucket", sourceFileSystem.bucket()); | ||
| assertEquals("new-bucket", destFileSystem.bucket()); | ||
| } catch (IOException e) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IOException should fail test; i.e. don't catch it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
dmitry-fa left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Your change does not fix the original issue which requests not creating a new CloudStorageFileSystem for a bucket and a provider, if such file system has been already created.
To achieve that you need to remember all created file systems and before creating a new one in the forBucket() method do search among previously created.
The code could look like:
private static final Map<String, CloudStorageFileSystem> fileSystemMap = new HashMap<>(); ... public static CloudStorageFileSystem forBucket( String bucket, CloudStorageConfiguration config, @Nullable StorageOptions storageOptions) { ... CloudStorageFileSystem existing = getExisitngCloudStorageFileSystem(bucket, config, storageOptions); if (existing == null) { exiting = new CloudStorageFileSystem (bucket, config, storageOptions); fileSystemMap.put(bucket, existing); } return existing; } | return new CloudStorageFileSystem( | ||
| cloudStorageFileSystem.provider(), bucketName, cloudStorageFileSystem.config()); | ||
| } | ||
| |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Method name getExistingCloudStorageConfiguration implies returning existing configuration, you return FileSystem and create it each time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
| this.config = config; | ||
| this.cloudStorageFileSystem = this; | ||
| } | ||
| |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this.cloudStorageFileSystem = this does not work as you probably expect. Each time a new CloudStorageFileSystem object is created the static field cloudStorageFileSystem will be overridden.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
f463c31 to 8811b6c Compare 1f3643b to d3c8b8b Compare | @dmitry-fa @frankyn ,PTAL |
...cloud-nio/src/test/java/com/google/cloud/storage/contrib/nio/CloudStorageFileSystemTest.java Outdated Show resolved Hide resolved
...cloud-nio/src/test/java/com/google/cloud/storage/contrib/nio/CloudStorageFileSystemTest.java Outdated Show resolved Hide resolved
dmitry-fa left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, thanks for addressing all my comments. Please wait for a review from a Googler
| @frankyn ,gentle ping |
🤖 I have created a release \*beep\* \*boop\* --- ## [0.122.0](https://www.github.com/googleapis/java-storage-nio/compare/v0.121.2...v0.122.0) (2020-11-06) ### Features * update cloudstorageconfiguration to improve copy accross cross-bucket performance ([#168](https://www.github.com/googleapis/java-storage-nio/issues/168)) ([db74524](https://www.github.com/googleapis/java-storage-nio/commit/db74524d68487df71c80b122d8c0ff384dc9ace3)) * **deps:** adopt flatten plugin and google-cloud-shared-dependencies ([#156](https://www.github.com/googleapis/java-storage-nio/issues/156)) ([510f6a5](https://www.github.com/googleapis/java-storage-nio/commit/510f6a5efc4a13b010020a8d67ec1511bbd46564)) ### Bug Fixes * FakeStorageRpc#list to filtering the files by bucket and prefix ([#208](https://www.github.com/googleapis/java-storage-nio/issues/208)) ([21f606e](https://www.github.com/googleapis/java-storage-nio/commit/21f606eca67b1c8a471ee28f7e2dd3851a0c493e)) * implement writeWithResponse in FakeStorageRpc ([#187](https://www.github.com/googleapis/java-storage-nio/issues/187)) ([10ddfae](https://www.github.com/googleapis/java-storage-nio/commit/10ddfae3923f1d5e64b151293db238f1af433d03)) ### Dependencies * update dependency com.google.apis:google-api-services-storage to v1-rev20200611-1.30.9 ([#166](https://www.github.com/googleapis/java-storage-nio/issues/166)) ([3cab5f2](https://www.github.com/googleapis/java-storage-nio/commit/3cab5f25e081c0198d06903a42c7de3dbd34bbad)) * update dependency com.google.apis:google-api-services-storage to v1-rev20200727-1.30.10 ([#171](https://www.github.com/googleapis/java-storage-nio/issues/171)) ([62998f0](https://www.github.com/googleapis/java-storage-nio/commit/62998f0adfb22b194e2dd633532e13bf27a79479)) * update dependency com.google.apis:google-api-services-storage to v1-rev20200814-1.30.10 ([#206](https://www.github.com/googleapis/java-storage-nio/issues/206)) ([0d7cd44](https://www.github.com/googleapis/java-storage-nio/commit/0d7cd44e73b0b9456a8ba96cc6e6d0e60a0b7d4e)) * update dependency com.google.apis:google-api-services-storage to v1-rev20200927-1.30.10 ([#233](https://www.github.com/googleapis/java-storage-nio/issues/233)) ([91b7918](https://www.github.com/googleapis/java-storage-nio/commit/91b7918539186763ecaa377cfb6f3908105df279)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.10.0 ([#226](https://www.github.com/googleapis/java-storage-nio/issues/226)) ([83acbd7](https://www.github.com/googleapis/java-storage-nio/commit/83acbd766b06d66fc9a41385fdd49aa57d9a0291)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.10.1 ([#239](https://www.github.com/googleapis/java-storage-nio/issues/239)) ([bd1928d](https://www.github.com/googleapis/java-storage-nio/commit/bd1928da234dae6fc2fb1ad6658d89a9ebc7af82)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.10.2 ([#243](https://www.github.com/googleapis/java-storage-nio/issues/243)) ([4ca9869](https://www.github.com/googleapis/java-storage-nio/commit/4ca9869d99fc3d2e578fb86fbd9053f855f1e51c)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.12.0 ([#246](https://www.github.com/googleapis/java-storage-nio/issues/246)) ([3e6171b](https://www.github.com/googleapis/java-storage-nio/commit/3e6171b2d4c4c7832b62288faf30f767c5278762)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.12.1 ([#255](https://www.github.com/googleapis/java-storage-nio/issues/255)) ([73d7dd2](https://www.github.com/googleapis/java-storage-nio/commit/73d7dd2ec6ca371480955c844ffb5e6aed4f608f)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.13.0 ([#260](https://www.github.com/googleapis/java-storage-nio/issues/260)) ([54895cf](https://www.github.com/googleapis/java-storage-nio/commit/54895cf441f1e1723cf941aa1f6cab249acae6b8)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.14.1 ([#271](https://www.github.com/googleapis/java-storage-nio/issues/271)) ([058d7c9](https://www.github.com/googleapis/java-storage-nio/commit/058d7c9c1471846f3d897cac31cb5f6d42ce5a7d)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.2 ([#167](https://www.github.com/googleapis/java-storage-nio/issues/167)) ([3b14bbc](https://www.github.com/googleapis/java-storage-nio/commit/3b14bbc05d5658c57d4f272fdfdf152d7d9ee18d)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.3 ([#169](https://www.github.com/googleapis/java-storage-nio/issues/169)) ([4e7bac1](https://www.github.com/googleapis/java-storage-nio/commit/4e7bac10c2466d1407e8f1a6de838261e6c1b097)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.4 ([#189](https://www.github.com/googleapis/java-storage-nio/issues/189)) ([af492b8](https://www.github.com/googleapis/java-storage-nio/commit/af492b8068101727793e64e89da2778205fa08b6)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.6 ([#191](https://www.github.com/googleapis/java-storage-nio/issues/191)) ([9bbc1fc](https://www.github.com/googleapis/java-storage-nio/commit/9bbc1fc755271f2ac6b798d50f6c0ba8e022c589)) * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.9.0 ([#202](https://www.github.com/googleapis/java-storage-nio/issues/202)) ([cf312a3](https://www.github.com/googleapis/java-storage-nio/commit/cf312a3fe46ec446d1b5ee418849b3ab9abee87e)) * update dependency com.google.cloud:google-cloud-storage to v1.110.0 ([#154](https://www.github.com/googleapis/java-storage-nio/issues/154)) ([fd6de38](https://www.github.com/googleapis/java-storage-nio/commit/fd6de38ee63376afff03c7e794dce1f4524a4375)) * update dependency com.google.cloud:google-cloud-storage to v1.112.0 ([#199](https://www.github.com/googleapis/java-storage-nio/issues/199)) ([8a38817](https://www.github.com/googleapis/java-storage-nio/commit/8a3881732551bcef1d76f76026a29071fc50dd43)) * update dependency com.google.cloud:google-cloud-storage to v1.113.0 ([#205](https://www.github.com/googleapis/java-storage-nio/issues/205)) ([068002e](https://www.github.com/googleapis/java-storage-nio/commit/068002e1145418ecb848ed066346d1fc6ec14807)) * update dependency com.google.cloud:google-cloud-storage to v1.113.1 ([#212](https://www.github.com/googleapis/java-storage-nio/issues/212)) ([acc1fe8](https://www.github.com/googleapis/java-storage-nio/commit/acc1fe8f23c5926000c7a869ca9a71372850c46a)) * update dependency com.google.cloud:google-cloud-storage to v1.113.2 ([#266](https://www.github.com/googleapis/java-storage-nio/issues/266)) ([9d2c11d](https://www.github.com/googleapis/java-storage-nio/commit/9d2c11d009394f3f3ffd1b115b97b350aa64d1b7)) * update dependency com.google.guava:guava to v30 ([#263](https://www.github.com/googleapis/java-storage-nio/issues/263)) ([4e81dab](https://www.github.com/googleapis/java-storage-nio/commit/4e81daba421e0e42695203c6fe74670b32a3f576)) * update dependency org.mockito:mockito-core to v3.4.4 ([#170](https://www.github.com/googleapis/java-storage-nio/issues/170)) ([3e06bd5](https://www.github.com/googleapis/java-storage-nio/commit/3e06bd5b07065acb1b41a03ebbb862487c046eeb)) * update dependency org.mockito:mockito-core to v3.5.10 ([#203](https://www.github.com/googleapis/java-storage-nio/issues/203)) ([33fdc31](https://www.github.com/googleapis/java-storage-nio/commit/33fdc315a5b4bccdebb13262cbc5868011abe444)) * update dependency org.mockito:mockito-core to v3.5.11 ([#214](https://www.github.com/googleapis/java-storage-nio/issues/214)) ([575c308](https://www.github.com/googleapis/java-storage-nio/commit/575c30882ab84e1b2b1ce34d5176b4f8688a8136)) * update dependency org.mockito:mockito-core to v3.5.7 ([#194](https://www.github.com/googleapis/java-storage-nio/issues/194)) ([8cc7616](https://www.github.com/googleapis/java-storage-nio/commit/8cc76166baf3f5f49223bb8eafc8704679c4d427)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please).
Fixes #60