0
$\begingroup$

I have a corpus of free form text (emails) and am trying to extract the highest degree (eg. High School, Bachelor, Master's, phD) from each of them. Is stemming the way to go? Or lemmatization? Note that the degree may not be mentioned at all.

Some example emails that I'll need to go through:

  1. Hi. My name is XXXXX. I have bachelor's in Math from UPenn and a Master's of Education from Stony Brook. Would love to work with XXXX!

  2. Dear XXXX, my name is XXXX and am interested in your company. Please email me for CV.

Improve this question
$\endgroup$
2
  • $\begingroup$ Do you have labels with the actual degree of each email? If no, you could devise some heuristic to guess it, but you would certainly not be able to evaluate the obtained results. You could label your corpus by hand, but I think it won't be easy because at least I have no idea of how to guess the degree of a person from a sentence unless it is explicitly mentioned... $\endgroup$ Commented Aug 19, 2017 at 12:41
  • $\begingroup$ Unfortunately, no. In the event that the degree isn't mentioned, then I'll just have another label "NA / null" - I just want to be able to extract as much as I can from the free form text. That said, I don't know how larger of a training set I should work with. Labeling 100,000 emails would suck my soul away... $\endgroup$ Commented Aug 20, 2017 at 16:23

2 Answers 2

Reset to default
1
$\begingroup$

I would have started with a regexp to find all degrees in the email and then compare them to take the highest one (you can order them) ex. Masters > Bachelor. This might be a little bit too easy but worth try.

Improve this answer
$\endgroup$
0
$\begingroup$

I would hand label a few hundred cases just to build a test dataset to check how well your method works. I think this is a clear case where building some rules based system will work better, build a set of rules that just look for specific keywords, match all the words to your rules and pick the maximum degree that you could find. After running your new method through your test set, you can see where the model made mistakes and because you made the rules yourself, you can see why it made a mistake and how you could fix it.

If you are set on a more automated approach, you could look into rules mining or character based classification, but you would likely need to label much more data.

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.