Presuming the stipulations I mentioned in my comment above (the two integers are distinct and we cannot have the same number on both hats):
B gets it at the second question. B knows that his number is an addition of two of the factors of 4. This means the only possibilities are 4 and 5. Since we are assuming we can't have the same number on both hats, the only possibility is 5.
EDIT:
Without restrictions:
A gets on on round 3. A knows his possible numbers are 4 and 6. He passes on question #1. B knows his possibilities are 4 and 5. He does not have enough info to eliminate one, so he passes as well. Now, if A's number was 6, B would realize on round 2. In that case, B's possibilities would have been 5 and 7, and had it been 7, A would have got it on round 1 (7 is prime and only has 1 and 7 as factors. A's number could only be 8 in that case), since A did NOT, then the only choice left for B was 5. Since this was not the case, A knows his number is not 6, and therefore can only be 4.