It might also be very worth your while to take a look at fine-tuning. It will allow you to combine the strength of a pre-trained network and the domain knowledge advantage of training a new one. stats.stackexchange.com/questions/331369/…

No free lunch, so it's hard to give a straight answer. But I would experiment with 'jaccard' and 'matching'. They are also in the docs, so should work like a charm.

I’m afraid that’s a completely different question: “Where does averaging occur in single-head attention mechanism X?” Like I said, I’m no expert on attention mechanisms, so I cannot answer your follow-up question without some in-depth research. But then it’s probably more efficient if you did that yourself.