I am using a U-Net to segment cancer cells in images of patients' arms. I would like to add patient data to it in order to see if it is possible to enhance the segmentation (patient data comes in the form of a table containing features such as gender, age, etc.). So far, my researches have led me nowhere. What can I do in order to achieve this?
$\begingroup$ $\endgroup$
Add a comment |
1 Answer
$\begingroup$ $\endgroup$
What you want to do is called multi-task learning. Here's what you do:
- Create a second Input.
- Attach it to 1D CNN (2-3 layers), so it aggregates this tabular information.
- Concatenate this feature with the intermediate feature generated by the U-Net using Concatenate layer.
- Put a dense layer of 2 after this.
- Put softmax with units = number of classes.
- Add CE loss calculated with this one and the ground labels to the loss of U-Net.
This is in regards to TensorFlow. The same can be done in PyTorch easily.
