ENH: Allow for join between two multi-index dataframe instances #20356

harisbal · 2018-03-15T01:04:14Z

closes #16162
closes #6360

Allow to join on multiple levels for multi-indexed dataframe instances

doc/source/merging.rst

doc/source/whatsnew/v0.23.0.txt

pandas/core/indexes/base.py

pandas/core/reshape/merge.py

pandas/tests/reshape/merge/test_merge.py

jreback · 2018-03-15T10:35:42Z

pandas/tests/reshape/merge/test_merge.py

+ 'LinkType', 'Distance'])
+ .set_index(['Origin', 'Destination', 'Period', 'LinkType']))
+
+ def f():


are there other error conditions to test?

This test fails and I'm not sure how we should handle join on empty levels

https://github.com/harisbal/pandas/blob/d694f254f34d3a8bb1ccdbafbc876d5f28ea3323/pandas/tests/reshape/merge/test_merge.py#L1429-L1446

can you show a mini-example?

@jreback Here is an example of a join on 2 multilevel indexed (same levels) dfs using two different methods

pd.merge(df1.reset_index(), df2.reset_index(),...)

df1.join(df2)

The results differ. Do you think that's an issue? I'm facing a similar issue when I try a multi-level join.

import numpy as np import pandas as pd join_type='left' left_multi=( pd.DataFrame( dict(Origin=['A', 'A', 'B', 'B', 'C'], Destination=[np.nan] * 5, Trips=[1987, 3647, 2470, 4296, 4444]), columns=['Origin', 'Destination', 'Trips']) .set_index(['Origin', 'Destination'])) right_multi=( pd.DataFrame( dict(Origin=['A', 'A', 'B', 'B', 'C', 'C', 'E'], Destination=[np.nan] * 7, Distance=[100, 80, 90, 80, 75, 35, 55]), columns=['Origin', 'Destination', 'Distance']) .set_index(['Origin', 'Destination'])) on_cols = ['Origin', 'Destination'] idx_cols = ['Origin', 'Destination'] expected = (pd.merge(left_multi.reset_index(), right_multi.reset_index(), how=join_type, on=on_cols).set_index(idx_cols) .sort_index()) result = left_multi.join(right_multi, how=join_type).sort_index() print(expected) print(result)

jreback · 2018-03-15T10:36:52Z

pandas/core/reshape/merge.py

+
+ # Inject -1 in the labels list where a join was not possible
+ # IOW indexer[i]=-1
+ labels = [restore_labels[i] if i != -1 else -1 for i in indexer]


this should be a set operation on the arrays i think

Sorry @jreback but I'm not sure what you mean

I think this was addressed here (Thanks to @TomAugspurger):

https://github.com/harisbal/pandas/blob/be862c757f1249132e8b686f37c2ad0cd05c2143/pandas/core/reshape/merge.py#L1195-L1198

codecov · 2018-03-15T16:43:21Z

Codecov Report

Merging #20356 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@ Coverage Diff @@ ## master #20356 +/- ## ========================================== + Coverage 92.24% 92.25% +<.01%  ========================================== Files 161 161 Lines 51339 51376 +37 ========================================== + Hits 47360 47397 +37  Misses 3979 3979

Flag	Coverage Δ
#multiple	`90.64% <100%> (ø)`	⬆️
#single	`42.31% <1.75%> (-0.03%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/reshape/merge.py	`94.24% <100%> (+0.23%)`	⬆️
pandas/core/indexes/base.py	`96.48% <100%> (+0.01%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e413c49...8b5d0aa. Read the comment docs.

pep8speaks · 2018-03-17T18:23:10Z

Hello @harisbal! Thanks for updating the PR.

There are no PEP8 issues in the file pandas/core/indexes/base.py !
There are no PEP8 issues in the file pandas/core/reshape/merge.py !
There are no PEP8 issues in the file pandas/tests/reshape/merge/test_merge.py !
There are no PEP8 issues in the file pandas/tests/reshape/merge/test_multi.py !

Comment last updated on November 11, 2018 at 04:31 Hours UTC

shenker · 2018-09-14T18:03:52Z

Any progress on this?

jreback · 2018-09-18T13:18:53Z

sorry me take a look. i know this has been outstanding for quite some time.

jreback

can you rebase and let's get this in

doc/source/whatsnew/v0.23.0.txt

pandas/tests/reshape/merge/test_merge.py

harisbal · 2018-09-18T14:06:52Z

I'll take a look asap. Cheers

harisbal · 2018-09-20T13:36:36Z

Any idea why pandas-dev.pandas failed?

jreback

looks good, @jorisvandenbossche @TomAugspurger if you'd have a look

pandas/core/indexes/base.py

pandas/core/reshape/merge.py

pandas/tests/reshape/merge/test_multi.py

doc/source/whatsnew/v0.24.0.txt

jreback

can you update some older questions

jreback · 2018-11-04T15:36:55Z

pandas/core/reshape/merge.py

+
+ Parameters
+ ----------
+ left : Index


can you update this

doc/source/whatsnew/v0.24.0.txt

jreback · 2018-11-04T15:37:30Z

pandas/core/reshape/merge.py

+ list of non-common levels
+ join_idx : Index
+ the index of the join between the common levels of left and right
+ lidx : intp array


can you update

jreback · 2018-11-04T15:37:39Z

pandas/core/reshape/merge.py

+
+ Returns
+ -------
+ levels : intp ndarray


is this correct?

pandas/core/reshape/merge.py

…ex-join # Conflicts: # doc/source/whatsnew/v0.24.0.txt # pandas/core/reshape/merge.py # pandas/tests/reshape/merge/test_multi.py

pandas/core/reshape/merge.py

pandas/tests/reshape/merge/test_multi.py

TomAugspurger · 2018-11-14T21:05:12Z

How's this looking? I haven't checked on the changes in a while, but CI is passing.

jreback · 2018-11-14T21:26:39Z

@TomAugspurger I had some more comments. let me have a look again.

jreback · 2018-11-14T21:28:46Z

@harisbal can you merge master

@TomAugspurger this lgtm. let's merge and can followup on any small issues.

…dex-join

TomAugspurger · 2018-11-14T21:41:21Z

Merged master. Ping on green.

harisbal · 2018-11-15T09:27:05Z

Shall I try to merge again?

TomAugspurger · 2018-11-15T11:58:38Z

I restarted that crashed worker. I haven't seen that failure before.

TomAugspurger · 2018-11-15T13:08:21Z

All green. Merging.

Thanks!

harisbal · 2018-11-15T13:13:23Z

@jreback @TomAugspurger @WillAyd Thank you so much for everything!!
Sincere apologies for putting you in so much trouble with this PR, I really appreciate your help.
Hopefully my next PR will be smoother.

…as-dev#20356)

harisbal changed the title ~~ENH: Allow for join between to multi-index dataframe instances~~ ENH: Allow for join between two multi-index dataframe instances Mar 15, 2018

harisbal force-pushed the multi-index-join branch from fa4dbac to d694f25 Compare March 15, 2018 01:08

jreback added Enhancement Reshaping Concat, Merge/Join, Stack/Unstack, Explode API Design and removed API Design labels Mar 15, 2018

jreback requested changes Mar 15, 2018

View reviewed changes

harisbal force-pushed the multi-index-join branch 2 times, most recently from 310bf7a to a6c9733 Compare March 17, 2018 18:22

harisbal force-pushed the multi-index-join branch from f668710 to 8e5fcf1 Compare March 17, 2018 22:16

jreback requested changes Sep 18, 2018

View reviewed changes

doc/source/whatsnew/v0.23.0.txt Show resolved Hide resolved

pandas/tests/reshape/merge/test_merge.py Outdated Show resolved Hide resolved

harisbal force-pushed the multi-index-join branch 2 times, most recently from de6c469 to 50c90cc Compare September 19, 2018 15:56

Allow for join between two multi-index dataframe instances

b581789

harisbal force-pushed the multi-index-join branch 2 times, most recently from 6bd10f4 to 5689f0a Compare September 19, 2018 16:14

Merge remote-tracking branch 'upstream/master' into multi-index-join

2d61a12

harisbal force-pushed the multi-index-join branch from 5689f0a to 2d61a12 Compare September 19, 2018 16:47

harisbal force-pushed the multi-index-join branch from db133f0 to 01ae19e Compare October 7, 2018 21:19

Merge remote-tracking branch 'upstream/master' into multi-index-join

4d4acc5

harisbal force-pushed the multi-index-join branch from 01ae19e to 4d4acc5 Compare October 7, 2018 21:51

jreback requested changes Oct 7, 2018

View reviewed changes

jreback reviewed Oct 7, 2018

View reviewed changes

doc/source/whatsnew/v0.24.0.txt Outdated Show resolved Hide resolved

harisbal force-pushed the multi-index-join branch from 47bb4fe to 8b44f42 Compare October 8, 2018 00:13

jreback requested changes Nov 4, 2018

View reviewed changes

harisbal added 2 commits November 4, 2018 17:49

Update v0.24.0.txt

be862c7

Update docstring _restore_dropped_levels_multijoin

e10cbde

harisbal force-pushed the multi-index-join branch from 6d6678b to 0e9c060 Compare November 4, 2018 20:32

harisbal and others added 2 commits November 4, 2018 22:41

Update docstring _restore_dropped_levels_multijoin

06d48d0

Merge remote-tracking branch 'upstream/master' into multi-index-join

f54c151

harisbal force-pushed the multi-index-join branch from 0e9c060 to f54c151 Compare November 4, 2018 22:44

harisbal added 4 commits November 5, 2018 10:32

Merge remote-tracking branch 'origin/multi-index-join' into multi-ind…

c75108d

…ex-join # Conflicts: # doc/source/whatsnew/v0.24.0.txt # pandas/core/reshape/merge.py # pandas/tests/reshape/merge/test_multi.py

Merge remote-tracking branch 'upstream/master' into multi-index-join

c690260

updated comments

4092b34

Refactoring

cfd5fcc

jreback requested changes Nov 8, 2018

View reviewed changes

pandas/core/reshape/merge.py Outdated Show resolved Hide resolved

pandas/tests/reshape/merge/test_multi.py Show resolved Hide resolved

pandas/tests/reshape/merge/test_multi.py Outdated Show resolved Hide resolved

harisbal added 2 commits November 10, 2018 16:58

Review

6c8131d

Merge remote-tracking branch 'upstream/master' into multi-index-join

ecaf515

harisbal force-pushed the multi-index-join branch from be4aec7 to ecaf515 Compare November 11, 2018 04:31

jreback approved these changes Nov 14, 2018

View reviewed changes

Merge remote-tracking branch 'upstream/master' into harisbal-multi-in…

8b5d0aa

…dex-join

TomAugspurger merged commit 88cbce3 into pandas-dev:master Nov 15, 2018

tm9k1 pushed a commit to tm9k1/pandas that referenced this pull request Nov 19, 2018

ENH: Allow for join between two multi-index dataframe instances (pand…

4d8af7c

…as-dev#20356)

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

ENH: Allow for join between two multi-index dataframe instances (pand…

79a9322

…as-dev#20356)

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

ENH: Allow for join between two multi-index dataframe instances (pand…

a2dd4f0

…as-dev#20356)

mroeschke mentioned this pull request Apr 3, 2020

Join on level between two UNIQUE MultiIndex objects with unique level broadcast #15417

Closed

Uh oh!

ENH: Allow for join between two multi-index dataframe instances #20356

ENH: Allow for join between two multi-index dataframe instances #20356

Uh oh!

Conversation

harisbal commented Mar 15, 2018 • edited by jreback Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harisbal Apr 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harisbal Nov 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

codecov bot commented Mar 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

pep8speaks commented Mar 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated on November 11, 2018 at 04:31 Hours UTC

shenker commented Sep 14, 2018

jreback commented Sep 18, 2018

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

harisbal commented Sep 18, 2018

harisbal commented Sep 20, 2018

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TomAugspurger commented Nov 14, 2018

jreback commented Nov 14, 2018

jreback commented Nov 14, 2018

TomAugspurger commented Nov 14, 2018

harisbal commented Nov 15, 2018

TomAugspurger commented Nov 15, 2018

TomAugspurger commented Nov 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

harisbal commented Nov 15, 2018

Labels

6 participants

harisbal commented Mar 15, 2018 •

edited by jreback

Loading

harisbal Apr 7, 2018 •

edited

Loading

harisbal Nov 4, 2018 •

edited

Loading

codecov bot commented Mar 15, 2018 •

edited

Loading

pep8speaks commented Mar 17, 2018 •

edited

Loading

TomAugspurger commented Nov 15, 2018 •

edited

Loading