DOC: Add documentation on parquet categorical data handling #58245
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
abeltavares force-pushed the BUG/Parquet-size-grows-exponential-for-categorical-data branch from 8a5fe98 to c74b597 Compare 
Contributor Author
| This PR is ready for review. Thanks. |
Member
| /preview |
Contributor
| Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/58245/ |
mroeschke reviewed Apr 15, 2024
pandas/core/frame.py Outdated
| categories, not just those present in the data. This behavior | ||
| is expected and consistent with pandas' handling of categorical data. | ||
| * To manage file size and ensure a more predictable roundtrip process, | ||
| consider using `remove_unused_categories` on the DataFrame before saving. |
Member
There was a problem hiding this comment.
Suggested change
| consider using `remove_unused_categories` on the DataFrame before saving. | |
| consider using :meth:`Categorical.remove_unused_categories` on the DataFrame before saving. |
mroeschke reviewed Apr 15, 2024
pandas/core/frame.py Outdated
Comment on lines 2886 to 2887
| is expected and consistent with pandas' handling of categorical data. | ||
| * To manage file size and ensure a more predictable roundtrip process, |
Member
There was a problem hiding this comment.
Suggested change
| is expected and consistent with pandas' handling of categorical data. | |
| * To manage file size and ensure a more predictable roundtrip process, | |
| is expected and consistent with pandas' handling of categorical data. | |
| To manage file size and ensure a more predictable roundtrip process, |
mroeschke added IO CSV abeltavares force-pushed the BUG/Parquet-size-grows-exponential-for-categorical-data branch from c74b597 to d5020e1 Compare mroeschke approved these changes Apr 19, 2024
Member
| Thanks @abeltavares |
pmhatre1 pushed a commit to pmhatre1/pandas-pmhatre1 that referenced this pull request May 7, 2024
…ev#58245) Co-authored-by: Abel Tavares <abel.tavares@ctw.bmwgroup.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.