Yes, it is possible to modify just the record detail page for a specific custom object, but you'll need to create that page first.
When you are in Community Builder, go to Page Manager > + > Create Object Page > [Choose your Object] > Create. Check the screenshot for the first 3 steps.
Now you can go back to Community Builder > Page Editor > OBJECT_NAME Detail and edit the record detail page layout for only this specific object.
Inside the Page Manager you can even create multiple Page Variations for specific pages, showing a different page variation depending on the user's profile.
