How to use a capacitor to maintain power when switching between car batteries

All due respect, you're making it harder than it is :)

This is a common and well-solved problem with plenty of solutions. Simplest is to just put the radio, lights etc. on the house battery and done. No relay.

How do you charge the house battery? With any of a variety of battery isolators made for the RV industry. These cleverly charge the house battery from the engine alternator or EV DC/DC converter when the car battery is topped up and the excess alternator/DC-DC power is surplus.

This does not conflict with other ways of sustaining the house battery, such as solar.

Is this the best solution?

It's a solution that is highly likely to work.

What type of capacitor should I use? (Obviously it needs to be at least 15 volts, but how do I calculate the capacitance, assuming the stereo draws, say, 50 watts, and what other details are important?)

50 watts at 12 volts implies a current of around 4 amps. That's the first thing. The second thing is the contact switching time and, this may be 10 ms. The third thing is how low the voltage can droop to when the relay is in the process of switching. This third thing depends on the stereo and, how low the voltage can drop to without it losing its settings. I'll take a guess at 7 volts (a drop of 5 volts from 12 volts).

The hold-up (electrolytic) capacitor is then calculated by this: -

$$C = \text{switching period} \times \text{current} \div \text{volt drop}$$

I get a value of 4 amps × 10 ms ÷ 5 volts = about 8 mF (8,000 uF)

Choose a voltage rating of 25 volts is my advice.

Should I worry about a flyback diode?

A flyback diode isn't necessary

It depends on the current being drawn from a battery at the time you want to make the switch. If the car is turned off then it should be quite small or else it would drain your car battery.Then a capacitance value can be calculated by:$$C\approx\frac{I\Delta t}{\Delta V}$$.

\$\Delta t\$ is the time it takes the relay to switch batteries.

\$\Delta V\$ is the maximum allowed voltage drop.

Starting the car pulls the voltage as low as 9V, but does not reset the radio. So \$\Delta V\$ could be as much as 3V.

Lets assume the relay takes 0.1 second to switch and 10mA. Then$$C\approx\frac{0.01\cdot0.1}{3}\approx330\mu F$$

I expect if the car is off then the current will be much lower. If the switch is in the accessory position it will be much higher.

A 1V or 2 V \$\Delta V\$ may be better but will need a bigger capacitor.

It is best to measure the current, then multiply in a safety margin.

Is this the best solution?

It can be a viable solution, but the effectiveness will depend on the exact specifications of your stereo system and the duration of the power loss during switching. The capacitor would need to supply power long enough to cover the switching time, and have enough capacitance to supply the necessary current without the voltage dropping too much.

What type of capacitor should I use?

You would likely want to use an electrolytic capacitor because they can provide a high capacitance in a small package and they are suitable for low-frequency (DC) applications like this.

The necessary capacitance can be roughly estimated by first determining the amount of charge (Q) needed. This can be calculated from the power (P) of the stereo and the time (t) of the power interruption as follows:

Q = P * t

Assuming the power interruption is very brief (say, 0.1 seconds), and the stereo is a 50W system running at 12V (which gives a current of about 4.2A), the required charge would be:

Q = 50W * 0.1s = 5 Joules

The energy stored in a capacitor can be calculated as:

E = 0.5 * C * V^2

Rearranging for C gives:

C = 2E / V^2

Substituting the values:

C = 2 * 5J / (12V)^2 = 0.07 Farads or 70,000 microFarads

So, a capacitor of around 70,000 microFarads (uF) and rated for at least 15V (preferably more for safety margin) should suffice.

Would any other modifications to the circuit be a good idea? e.g. Should I worry about a flyback diode?

Including a flyback diode could be a good idea to prevent any potential reverse voltage spikes which could occur when the relay is switched off. This is particularly important if you have other sensitive electronics connected to the same circuit.