Skip to content

fix: add more native image configurations for Arrow tests and enable native image tests #2053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
May 25, 2022
Merged

fix: add more native image configurations for Arrow tests and enable native image tests #2053

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
3059e49
fix: add more native image configurations for Arrow tests
mpeddada1 May 18, 2022
1030790
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 18, 2022
7502c2d
add more necessary configurations
mpeddada1 May 19, 2022
4612366
Merge branch 'fix-arrow-tests' of github.com:googleapis/java-bigquery…
mpeddada1 May 19, 2022
fde4c0b
enable native image tests
mpeddada1 May 19, 2022
463c852
empty commit
mpeddada1 May 20, 2022
f64545d
Merge branch 'main' of github.com:googleapis/java-bigquery into fix-a…
mpeddada1 May 20, 2022
db501a8
remove exclusion
mpeddada1 May 23, 2022
6cd1d3b
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 23, 2022
b95d622
Merge branch 'main' of github.com:googleapis/java-bigquery into fix-a…
mpeddada1 May 24, 2022
b2c67c6
exclude arrow tests in java 17
mpeddada1 May 24, 2022
a050229
Merge branch 'fix-arrow-tests' of github.com:googleapis/java-bigquery…
mpeddada1 May 24, 2022
7ac06da
restore nightly-it
mpeddada1 May 24, 2022
2295f7b
restore formatting
mpeddada1 May 24, 2022
ca77139
more formatting changes
mpeddada1 May 24, 2022
59e72c0
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 24, 2022
361fe4a
Merge branch 'fix-arrow-tests' of https://github.com/googleapis/java-…
gcf-owl-bot[bot] May 24, 2022
ead0f30
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 24, 2022
ab15abf
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 24, 2022
ffddacd
Merge branch 'fix-arrow-tests' of https://github.com/googleapis/java-…
gcf-owl-bot[bot] May 24, 2022
8db8fd1
include tests for exclusion
mpeddada1 May 24, 2022
57f9d9d
bring back owlbot exclusion
mpeddada1 May 24, 2022
b87ca45
restore script changes
mpeddada1 May 24, 2022
fc1e5fc
bring back nightly-it
mpeddada1 May 24, 2022
62dcfed
include IT* and *ClientTest filter from shared-config
mpeddada1 May 24, 2022
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions .kokoro/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,16 @@ nightly-it)
verify
RETURN_CODE=$?
;;
#graalvm)
# # Run Unit and Integration Tests with Native Image
# mvn -B ${INTEGRATION_TEST_ARGS} -ntp -Pnative -Penable-integration-tests test
# RETURN_CODE=$?
# ;;
graalvm)
# Run Unit and Integration Tests with Native Image
mvn -B ${INTEGRATION_TEST_ARGS} -ntp -Pnative -Penable-integration-tests test
RETURN_CODE=$?
;;
graalvm17)
# Run Unit and Integration Tests with Native Image. Skip Arrow tests until https://github.com/googleapis/java-bigquery/issues/2060 is fixed.
mvn -B ${INTEGRATION_TEST_ARGS} -ntp -Pnative -Penable-integration-tests test "-Dtest=!com.google.cloud.bigquery.it.ITBigQueryTest#testBQResultSetPaginationSlowQuery+testReadAPIConnectionMultiClose+testReadAPIIterationAndOrder, !com.google.cloud.bigquery.it.ITNightlyBigQueryTest#testIterateAndOrder+testMultipleRuns+testIterateAndOrderDefaultConnSettings, IT*, *ClientTest"
RETURN_CODE=$?
;;
samples)
SAMPLES_DIR=samples
# only run ITs in snapshot/ on presubmit PRs. run ITs in all 3 samples/ subdirectories otherwise.
Expand Down
59 changes: 59 additions & 0 deletions ...esources/META-INF/native-image/com.google.cloud/google-cloud-bigquery/reflect-config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,67 @@
"name":"io.netty.buffer.AbstractByteBufAllocator",
"queryAllDeclaredMethods":true
},
{
"name":"io.netty.buffer.PooledByteBufAllocator",
Copy link
Contributor

@stephaniewang526 stephaniewang526 May 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you intend to manually maintain this config file in the future? How much ongoing maintenance work is needed? Would we need to make these changes if we made code changes related to Arrow or add new Arrow dependencies? cc @prash-mi
Copy link
Contributor Author

@mpeddada1 mpeddada1 May 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good question. Given that we have a bit more background on GraalVM at the moment, we plan to maintain these configurations with client library owners being involved in the review process. However, we're currently working on making our onboarding documentation more robust as we learn more about the technology so that it is easier for client library owners to contribute and get unblocked sooner if needed in the future . Will keep you posted on the plans for that.

Would we need to make these changes if we made code changes related to Arrow or add new Arrow dependencies?

Most of these changes involved configurations for the netty classes that Apache Arrow brings in. These currently exist in gax-grpc but have been added here explicitly to avoid adding a dependency on gax-grpc. Since bigquery and most libraries bring in gax, we might want to consider moving them to gax or even closer to netty which could help make this file a little shorter. As far as the Arrow dependencies are concerned, I don't have a solid answer for this but there is a possibility that we'll need to add new configurations if new deps are added. Luckily, we have our Kokoro GraalVM presubmit checks to catch them if required.
"fields":[{"name":"directArenas"}]
},
{
"name":"io.netty.util.internal.shaded.org.jctools.queues.MpscChunkedArrayQueue",
"queryAllDeclaredMethods":true,
"allDeclaredFields":true,
"queryAllDeclaredConstructors":true
},
{
"name":"io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueConsumerIndexField",
"fields":[{"name":"consumerIndex"}]
},
{
"name":"io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerIndexField",
"fields":[{"name":"producerIndex"}]
},
{
"name":"io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerLimitField",
"fields":[{"name":"producerLimit"}]
},
{
"name":"java.nio.DirectByteBuffer",
"methods":[{"name":"<init>","parameterTypes":["long","int"] }]
},
{
"name":"org.apache.arrow.memory.DefaultAllocationManagerFactory",
"fields":[{"name":"FACTORY"}]
},
{
"name":"org.apache.arrow.vector.types.pojo.ArrowType",
"allDeclaredFields":true,
"queryAllDeclaredMethods":true,
"queryAllDeclaredConstructors":true
},
{
"name":"org.apache.arrow.vector.types.pojo.ArrowType$Int",
"allDeclaredFields":true,
"queryAllDeclaredMethods":true,
"queryAllDeclaredConstructors":true
},
{
"name":"org.apache.arrow.vector.types.pojo.ArrowType$PrimitiveType",
"allDeclaredFields":true,
"queryAllDeclaredMethods":true
},
{
"name":"io.netty.buffer.AbstractReferenceCountedByteBuf",
"fields":[{"name":"refCnt"}]
},
{
"name":"io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueColdProducerFields",
"fields":[{"name":"producerLimit"}]
},
{
"name":"io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueConsumerFields",
"fields":[{"name":"consumerIndex"}]
},
{
"name":"io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueProducerFields",
"fields":[{"name":"producerIndex"}]
}
]
2 changes: 1 addition & 1 deletion owlbot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,5 @@
'.kokoro/dependencies.sh',
'codecov.yaml',
'renovate.json',
'.kokoro/build.sh',
'.kokoro/build.sh'
Copy link
Contributor

@stephaniewang526 stephaniewang526 May 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be removed, right?
Copy link
Contributor Author

@mpeddada1 mpeddada1 May 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to bring this back in to exclude the arrow tests in Java 17 until #2060 is resolved.
])