Brtdku

I have a set of 3000 text documents and I want to extract top 300 keywords (could be single word or multiple words).

I have tried the below approaches -

RAKE: It is a Python based keyword extraction library and it failed miserably.

Tf-Idf: It has given me good keywords per document, but we I not able to aggregate them and find keywords that represent the whole group of documents.
Also, just selecting top k words from each document based on Tf-Idf score won't help, right?

Word2vec: I was able to do some cool stuff like find similar words but not sure how to find important keywords using it.

Can you please suggest some good approach (or elaborate how to improve any of the above 3) to solve this problem? Thanks :)

Is better for you to choose manually those 300 words (it's not so much and is one time) - Code Written in Python 3

import os

files = os.listdir()

topWords = ["word1", "word2.... etc"]

wordsCount = 0

for file in files: 

        file_opened = open(file, "r")

        lines = file_opened.read().split("n")

        for word in topWords: 

                if word in lines and wordsCount < 301:

                                print("I found %s" %word)

                                wordsCount += 1

        #Check Again wordsCount to close first repetitive instruction

        if wordsCount == 300:

                break

Most easy and effective way to apply the tf-idf implementation for most important words. if you have stop word you can filter the stops words before apply this code. hope this works for you.

import java.util.List;



/**

 * Class to calculate TfIdf of term.

 * @author Mubin Shrestha

 */

public class TfIdf {



    /**

     * Calculates the tf of term termToCheck

     * @param totalterms : Array of all the words under processing document

     * @param termToCheck : term of which tf is to be calculated.

     * @return tf(term frequency) of term termToCheck

     */

    public double tfCalculator(String totalterms, String termToCheck) {

        double count = 0;  //to count the overall occurrence of the term termToCheck

        for (String s : totalterms) {

            if (s.equalsIgnoreCase(termToCheck)) {

                count++;

            }

        }

        return count / totalterms.length;

    }



    /**

     * Calculates idf of term termToCheck

     * @param allTerms : all the terms of all the documents

     * @param termToCheck

     * @return idf(inverse document frequency) score

     */

    public double idfCalculator(List allTerms, String termToCheck) {

        double count = 0;

        for (String ss : allTerms) {

            for (String s : ss) {

                if (s.equalsIgnoreCase(termToCheck)) {

                    count++;

                    break;

                }

            }

        }

        return 1 + Math.log(allTerms.size() / count);

    }

}

import os

import operator

from collections import defaultdict

files = os.listdir()

topWords = ["word1", "word2.... etc"]

wordsCount = 0

words = defaultdict(lambda: 0)

for file in files:

    open_file = open(file, "r")

    for line in open_file.readlines():

        raw_words = line.split()

        for word in raw_words:

            words[word] += 1

sorted_words = sorted(words.items(), key=operator.itemgetter(1))

now take top 300 from sorted words, they are the words you want.