What I can't work out is the value of the smoothing capacitor I'll need. If it's too small I think I'll get 50Hz drops in the output voltage which will upset the down stream circuit.
That trouble is called "Ripple". When it cause problem to down stream, than let's name it "Ripple Effect", a new terminology now on.
Too high and could I get trouble charging it?
Yes, that trouble is called "surge current" through diode. What happens is that the "Voltage Source" tries to charge the capacitor with all possible force, and only limits are hidden (from the schematics) impedance inherent to the components. When that current goes over the "rated maximum" of the parts involved in the current path (transformer, rectifier, capacitors), that will harm the corresponding parts. These are all in the datasheet.
Is there a formulae or even a rule of thumb that will give me the right value?
Yes. The formulae is "ripple voltage calculation formula", let's name it, if it did not have name before. The idea is finding the degree of smoothing effect of the capacitor by specifying tolerable amount of the voltage variation on the load.
The input variables to the formula are: a. Input AC voltage, b. Load current, c. The ripple tolerance on the load.
The formula, to approximate the cap, is: dV = 1/C x I x dt
Example:
Trans former output = 9V AC
Rectifier drop = 1V
Load resistance = 100 ohm
Tolerable ripple = 0.5V
1) Peak voltage at the load side of the bridge rectifier
= Vac x root(2) - 2 x Vrect = 9V * 1.4 - 2 x 1V = 10.6V
2) The load current, Iload = Vpeak / Rload = 10.6V / 100 = 106mA
3) C = {Iload x [1 / (line_frequency x 2_for_full_bridge)]} / Vripple
= {106mA x [1 / (50Hz x 2)]} / 0.5V = 0.00212F | 10.6V => 2,200uF | 15V
The same formula is used as well to calculate the input capacitor value between rectified AC and a voltage regulator.