How do I diagnose the problems with this amplifier?

The first thing I like to do is to redraw the schematic. In the process of redrawing it, I also work to understand each section as I go.

output stage

Since the output power is the primary goal, I would start first with what I recognize as a 2-quadrant class-AB output stage:

^{simulate this circuit – Schematic created using CircuitLab}

It's just two parts. But already there's a few things to stop and consider.

The present circuit can deliver 10W of output power when driving an 8 Ohm load, or about 18W @ 4 Ohm.

So the output spec is \$10\:\text{W}\$ into \$8\:\Omega\$ or \$18\:\text{W}\$ into \$4\:\Omega\$. I can work out that the peak voltage excursion must be \$V_{\small{PEAK}}=\sqrt{2\cdot 4\:\Omega\cdot 18\:\text{W}}= 12 \:\text{V}\$ (the RMS voltage is \$V_{\small{RMS}}\approx 8.5\:\text{V}\$) or else \$V_{\small{PEAK}}=\sqrt{2\cdot 8\:\Omega\cdot 10\:\text{W}}= 12.6 \:\text{V}\$ (the RMS voltage is \$V_{\small{RMS}}\approx 8.9\:\text{V}\$.)

The peak current into the load must be \$I_{\small{PEAK}}=\sqrt{\frac{2\cdot 18\:\text{W}}{4\:\Omega}}\approx 3.0 \:\text{A}\$ into \$4\:\Omega\$ (the RMS current is \$I_{\small{RMS}}\approx 2.1\:\text{A}\$) or \$I_{\small{PEAK}}=\sqrt{\frac{2\cdot 10\:\text{W}}{8\:\Omega}}\approx 3.0 \:\text{A}\$ into \$8\:\Omega\$ (the RMS current is \$I_{\small{RMS}}\approx 1.1\:\text{A}\$).

So already I have my first set of sanity checks.

Including now the datasheets for the BDX53A and BDX54A (using Mouser here):

• The referenced schematic shows \$T_1\$ as a \$48\:\text{V}\$ CT transformer with \$\ge 20\:\text{VA}\$ rating. The arrangement of the DC power supply appears consistent with the goals.
• The VA rating of a transformer is the total apparent power, which involves both the primary and secondary. In the case of full-wave center-tapped arrangement, there is a an additional \$\sqrt{2}\$ factor applied to the secondary portion. The schematic specifies from \$20\:\text{VA}\$ to \$30\:\text{VA}\$. So, assuming a good quality transformer with a 5% regulation spec, I can expect about \$\frac{20\:\text{VA}}{\left(1+\sqrt{2}\right)\cdot\left(1-5\%\right)}\approx 8.7\:\text{W}\$. This doesn't meet the spec. So let's try their other number: \$\frac{30\:\text{VA}}{\left(1+\sqrt{2}\right)\cdot\left(1-5\%\right)}\approx 13\:\text{W}\$. That also doesn't meet the spec.
• Doubling the worst case \$V_{\small{PEAK}}\$, and accounting something for circuit overhead, I'd know that the power supply rail must not fall below \$\approx 30\:\text{V}\$ and likely should be higher. In round numbers, I'd also expect a loaded DC rail of about \$30\:\text{V}\$. This may be cutting things a little close. But I've been conservative so far. So perhaps this is fine.
• \$Q_5\$ and \$Q_6\$ must be able to more than support the required currents. The referenced schematic indicates that the two output power transistors are sufficient and consistent with the goals.
• \$Q_5\$ and \$Q_6\$ must be within their safe operating areas. They seem to be.

I'd also make a note to myself to verify that the DC rail meets the above, if and when the circuit is actively under test. Especially because of the transformer issues mentioned above and when under full loading.

An additional note as I move forward:

• Their minimum \$h_{\small{FE}}=750\$, with the typical value being about \$3\times\$ more. So the base current into these two transistors will not have to exceed a peak of \$3\:\text{mA}\$ but may only require \$1\:\text{mA}\$, in practice.

The biasing section, including the VBE multiplier, must be consistent with this requirement.

biasing section

Now I'd add in the biasing section from the schematic and add in the speaker output:

^{simulate this circuit}

There's the usual VBE multiplier (but lacking any Early Effect compensation) and the commonly found use of a bootstrap capacitor to help turn \$R_{11}\$ into a bit of a cheap current source. There's also the usual VAS stage, \$Q_3\$, but lacking the usual Miller cap between its base and collector.

Let's estimate the DC rail as \$30\:\text{V}\$. Then the shared emitter node of the circuit should be biased to about half that, or \$15\:\text{V}\$. The datasheets suggest a worst case \$2.5\:\text{V}\$ for either Darlington, so figure about \$17.5\:\text{V}\$ at the base of \$Q_5\$.

At this point, I can estimate a quiescent current of \$\frac{30\:\text{V}-15\:\text{V}-2.5\:\text{V}}{1.5\:\text{k}\Omega+4.7\:\text{k}\Omega}\approx 2.0\:\text{mA}\$. But \$3\:\text{mA}\$ was the conservative figure, although \$1\:\text{mA}\$ may be acceptable.

Then I add in some stuff that's a little odd:

^{simulate this circuit}

Note that I've added and highlighted a section of weird biasing, with NFB on the right side.

Up to this point, just these modest surprises: (1) a transformer that may be under-spec'd but is otherwise perfectly normal-looking; and, (2) a VBE multiplier current that may be only a little bit shy of a more conservative value, but not in much of a concerning way.

But now there's this moment when everything comes to a tight wad of: (3) DC biasing for \$Q_3\$; and, (4) some kind of frequency dependent AC negative feedback; and, (5) \$Q_3\$ driven by a JFET source follower via a potentiometer.

It all happens in a collision of sorts, a Gordian Knot, right here.

That's where some of the power from walking through a schematic, redrawing and extracting familiar topologies a step at a time, may be found. It helps uncover/expose residual unfamiliar bits, making them manifest and demanding more focused attention.

Exposed like this, I feel \$Q_3\$, as a BJT VAS stage, should be driven through high \$g_m\$ and not as it is being handled from the JFET pre-amplifier stage. And I'd use a PNP transistor for this purpose. This extra transistor would buy separation of purpose and some clarity.

This doesn't mean losing the JFET stage. I believe it's very high input impedance is necessary for guitar pickups, as this Ovation Breadwinner 1251-0-350D schematic, for example, suggests. So I'm not suggesting that you replace that part of your schematic. Keep it.

But coming back to the point, this is why the process of redrawing is a habit you should cultivate and develop and then sustain as a matter of routine practice.

So I'm only going to focus on the power output stage, with fixed-voltage gain, and add that PNP transistor I mentioned while separating out the functions a bit.

using what works and building from there

I'd like to start out by again noting that \$Q_3\$ is a typical VAS that should be driven with high \$g_m\$. A simple PNP topology is all I need, shown below centered on \$Q_1\$. \$Q_1\$ will work as a kind of comparator, comparing the output signal with the input signal, while providing the high \$g_m\$ I believe I want:

^{simulate this circuit}

In shaping some choices around \$Q_1\$, arranging things so that the sign of \$R_2\$'s current doesn't change may help to ensure that \$Q_1\$'s base-emitter voltage doesn't vary too much. This means that I don't want the voltage at \$Q_1\$'s emitter to be higher than the output swing at the power emitter side of \$C_{12}\$. I can do that. But only by setting \$Q_1\$'s base voltage to about \$1\:\text{V}\$. Luckily, \$Q_3\$'s collector will be only one VBE (about \$660\:\text{mV}\$) above ground. (\$Q_1\$ will also not experience Early Effect in this arrangement.) So this idea may work.

\$Q_3\$'s base current will be \$\le 12\:\mu\text{A}\$, using a worst case estimate of \$h_{\small{FE}}\approx 250\$. So I want a large factor, perhaps \$10\times\$ that much as \$Q_1\$'s collector current, knowing that whatever \$Q_3\$'s base doesn't require can be sunk through resistor \$R_1\$ (which should be adjusted later in order to set the quiescent voltage of the node shared by the power transistor emitters.)

Let's call it \$100\:\mu\text{A}\$ as the PNP's collector current. This means that \$R_1=\frac{660\:\text{mV}}{100\:\mu\text{A}-12\:\mu\text{A}}= 7.5\:\text{k}\Omega\$. (I may adjust this value in the schematic, later, to set that quiescent voltage I just mentioned.)

I'll keep \$Q_1\$'s base at about \$1\:\text{V}\$. I've set up a divider with three resistors to achieve that. \$Q_1\$'s base current should be no more than \$1\:\mu\text{A}\$, so a divider with \$30\:\mu\text{A}\$ should be more than stiff enough.

\$R_2=\frac{15\:\text{V}-1\:\text{V}-600\:\text{mV}}{100\:\mu\text{A}+12\:\mu\text{A}}\approx 120\:\text{k}\Omega\$. And to get a voltage gain of about \$100\times\$ (you can adjust this voltage gain per your needs), I've set \$R_3=1.2\:\text{k}\Omega\$.

I've salted the schematic with similarly large values for added capacitors.

When I create the schematic, I'll make two adjustments:

1. Adjust potentiometer \$R_9\$ to achieve about \$30\:\text{mA}\$ (per the article) as a quiescent current for the power transistors.
2. Adjust \$R_1\$ to center the shared emitter node of the power transistors to \$15\:\text{V}\$.

Putting it into the schematic, I set the potentiometer and then found I needed to set \$R_1=6.8\:\text{k}\Omega\$, as well.

I'm expecting a voltage gain of slightly less than 100. With an input signal of \$120\:\text{mV}\$ I should see almost \$12\:\text{V}\$ peak at the output. I don't expect to get exactly \$\vert A_v\vert=100\$. But it should be close.

Let's see:

That worked well given a \$4\:\Omega\$ load. A gain of about 95.

Let's try it for \$8\:\Omega\$:

A gain of 97, now.

I'd say that will work. It has an advantage of building on what you have, too.

The volume control may be a potentiometer at the input or in the negative feedback chain. (That's either \$R_2\$ or \$R_3\$ or a creative use of a potentiometer between the two. I've not thought it through, but that's one possible place where I'd look. The other would be at the input, itself.)

final thoughts

I had mentioned the Miller compensation cap on \$Q_3\$ (and provided a floating capacitor labeled as \$C_7\$ in the earlier schematic.) Let's first look at the above schematic's AC analysis (without it) to see why:

There's very high gain out to about \$1\:\text{MHz}\$. And at \$0\:\text{dB}\$ the phase shows at about \$-215^\circ\$. Not good.

The Miller compensation goes in.

And I'd also included earlier another floating capacitor, \$C_2\$. That will help by dramatically reducing the voltage gain at high frequencies and on improving the phase margin.

I'll make both values \$33\:\text{pF}\$ and see where that goes:

At \$0\:\text{dB}\$ the phase shows at about \$-91^\circ\$ now and the upper corner is nicely placed at about \$20\:\text{kHz}\$. That may have a better chance of avoiding unwanted oscillations.

In pulling all this together, while keeping your JFET pre-amplifier, keep in mind also that I set the fixed voltage gain to an arbitrary value. Depending upon your JFET pre-amplifier, you may want to adjust this fixed voltage gain. This is done by changing the value of \$R_3\$ (not \$R_2\$). Larger values will reduce the fixed voltage gain. Smaller values will increase it. I'd recommend lowering the gain to the lower value you can accept. It's better that way.