10
$\begingroup$

NB: I am aware of the DeleteStopwords symbol.

I want to remove from a list of words the ones that are likely to be more banal. So for instance if we have

{"exegesis", "mystification", "bread", "synthesis", "dog", "autonomy", "develop", "enthusiastic", "house"}

then I would like a function that is likely to remove "bread", "dog", and "house" while leaving the other words in the list.

Any ideas for implementing such a thing? It doesn't have to be incredibly robust; I'm generating lists of words for games where the object is to guess the word.

Please add any useful tags; I couldn't think of any that applied very well.

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Why not create a list of words that you consider banal, then remove them with DeleteCases[] or something similar. Also, maybe define what is "banal" in this context? If the game was about dogs or food, surely 'house' and 'dog' would not be banal. $\endgroup$ Commented May 10, 2020 at 0:12
  • 3
    $\begingroup$ Might WordFrequencyData[] be a starting point for a simple commonality "score"? $\endgroup$ Commented May 10, 2020 at 0:23

2 Answers 2

Reset to default
11
$\begingroup$

Please also see a related problem here, it might give you more ideas. Words can be banal wrt all words in the dictionary or wrt some particular list of words. Let's consider the later case. WordFrequencyData gives the frequency of word in typical published English text. Within a given list of words you can find the max-frequency word and rescale the rest of the words according to it. Then a threshold to cut off more popular words will be always between 0 and 1.

banal[w_,t_]:=With[{freq=WordFrequencyData[w]}, Keys[Select[freq/Max[freq],#<=t&]]]

For your set of words:

words={"exegesis", "mystification", "bread", "synthesis", "dog", "autonomy", "develop", "enthusiastic", "house"};

and threshold $0.1$ we get

In[]:= banal[words,.1] Out[]= {exegesis,mystification,synthesis,autonomy,enthusiastic}

with the dropped words being:

In[]:= Complement[words,%] Out[]= {bread,develop,dog,house}
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Fantastic answer! I was unable to discover WordFrequencyData on my own and it is precisely the sort of symbol I was after. $\endgroup$ Commented May 10, 2020 at 17:10
0
$\begingroup$

Whether a word is "banal" is a subjective judgment. If you want to approach it in a statistical way, I'd try a TF/IDF approach. You'll find no shortage of references about it on the web.

Here's the basic idea. Suppose you're building a search index over a bunch of documents.

TF = term frequency: How many times a word appears in one particular document

IDF = inverse document frequency: 1 over the number of documents where the word appears. So, the more documents the word appears in, the smaller this factor becomes. 1 over 10 documents is a bigger number than 1 over 100,000 documents.

You can think of this roughly as a generalization of the stopword concept. Rather than manually list out your stopwords, you use automated means to count words over a set of documents and come up with what you might think of as a distinctiveness score. The word "the" might appear many times within one document, but it also appears across many documents, so it's not a very distinctive word.

So suppose you're building a system like Google, and the user searches for "the green xylophone". You probably don't want to rank documents very high just because they contain "the". However, "xylophone" is a more distinctive word in the sense that it appear in far fewer documents, so you might consider a match on that word to be more important in ranking a document up. "green" would be somewhere in the middle, probably.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ I see you're a new contributor, welcome! Do you have some Wolfram Language code for this idea? It sounds very neat & well thought out. It would be great if you could show us how to do what you're suggesting here :D $\endgroup$ Commented May 10, 2020 at 13:48

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.