Error: List controllers are not supported for AccountContactRelation

Unfortunetly, the AccountContactRelation object does not support the StandardSetController class.

Here is a workaround I used. You cannot override the "Add Relationship" button but you can hide it and add your own custom list view buttons.

1. Create a new list view button for the Account Contact Relationship Object. Log into Salesforce > Setup > Search for "Account Contact Relationships" > Click Buttons, Links and Actions > Click New Button or Link > Create a List Button with a Content Source of URL. Enter the URL address of your visualforce page. In my example my visualforce page is called AccountContactNew so I added the url /apex/AccountContactNew. I also added the account id as a URL parameter. (see screenshot for example)

2. Edit the Account detail page layout. Add the Related Contacts related list, if its not already added. While still editing the layout click the wrench icon on the Related Contacts related. Expand the Buttons section and deselect any standard buttons you want to hide. Add your custom list view button to the Selected Buttons List. Click OK. Save the layout. You should now see your new custom button on the Related Contacts related list.

I hope that helps!