1
$\begingroup$

I am doing a tweet clustering using DBSCAN algorithm. I use all the preprocessing steps and convert sentences to vector format before applying the algorithm. However, It always puts a lot of tweets in to the '0' class. The following is the plot showing eps with number of clusters.

enter image description here

The following are the parameters that I pass.

dbscan = DBSCAN(eps=0.15, min_samples=2, metric='cosine').fit(x)

The following are the resulting clusters.

enter image description here

label -1 1221 0 1349 1 2 2 2 3 4 ... 67 3 68 3 69 2 70 2 71 2

What is the reason that class 0 getting a high number of tweets than any other classes?

Improve this question
$\endgroup$
5
  • $\begingroup$ Can you please share some more insight on how you are processing the words before clustering. From initial observation all your clusters might be having some word that results in the cluster. Using word2vec embeddings and Euclidean might help $\endgroup$ Commented Nov 7, 2020 at 7:39
  • $\begingroup$ you need to tell us about how you converted tweets into vectors. That is the key part $\endgroup$ Commented Nov 7, 2020 at 12:55
  • $\begingroup$ @mahesh ghanta: Thanks, I have used Bag of Words, TFIDF, Spacy Vectors and also, Word2Vec. All produce the cluster No '0' with a large number of results. $\endgroup$ Commented Nov 8, 2020 at 9:20
  • $\begingroup$ @Kasra Manshaei: Thank you. Please see the previous comment. $\endgroup$ Commented Nov 8, 2020 at 9:20
  • $\begingroup$ Did you check the words that are key are important in this cluster ? Do they make sense? Also could you increase the neighbouring samples to atleast 5? $\endgroup$ Commented Nov 8, 2020 at 11:52

1 Answer 1

Reset to default
1
$\begingroup$

Two things: eps and quantitative representation of text.

You see that there is only for eps=0.15 a lot of clusters. But for others a lot less. This is hyper parameter that needs to be optimised (and min_samples)

And the other thing thats more important is what you use quantitative representation of text. You said Bag of Words, TFIDF, Spacy Vectors and also, Word2Vec, but did you tune them? DId you tree embeddings etc etc. There is a lot of improvement here, and when its good dbscan will function a lot better.

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.