-------------------------------------------------- | INCLUDED PROJECTS | -------------------------------------------------- *punctuation_detector A keras implementation of a period punctuation detector. This system inserts a period to a stream of unstructured text. *topic_based_segmentation A keras implementation of a system that divides a stream of sentences into segments where adjacent segments are different in topics. *segment_classification A kNN implementation (based on Word Embeddings and TFIDF features) that classifies a given segment into one of the available topics given their summaries. -------------------------------------------------- | DATASET USED | -------------------------------------------------- TDT-2 Corpus -------------------------------------------------- | DEPENDANCIES | -------------------------------------------------- keras 1.2.0 gensim word2vec numpy nltk -------------------------------------------------- | CORPUS | -------------------------------------------------- For the experiments, a TDTCorpus pickled object (*.pckl) is loaded that has two attributes: "text_corpus_bnds" and "sent_boundaries". Any object having these two attributes can work as input Explaining "text_corpus_bnds": A dict() object having as keys the ids of the documents, and values the list of tokens (words) of each document. Tokens (words) that mark an end of sentence contain a marker "<bnd>" EXAMPLE: text_corpus_bnds =	{	"document_id_1":["This","is","a", "sentence.<bnd>", "this", "is", "another", "sentence<bnd>"]	,	"document_id_2":["This","is","A", "non-pre-procces", "sentence.<bnd>", "this", "is", "another", "sentence<bnd>"]	,	"document_id_3":["This","is","a", "sentence.<bnd>", "this", "is", "another", "sentence<bnd>"]	} Explaining "sent_boundaries": A dict() object having as keys the ids of the documents, and values the list of sentence-based TOPIC boundaries. So each "i"th index in the list represents whether there is a topic change after "i"th sentence in the document or not. EXAMPLE: sent_boundaries =	{	"document_id_1":[0,0,0,1,0,0,0,0,0,1]	#document id 1 has two story segments: segment from sentence 1 > 4, segment from sentence 5 > 10	,	"document_id_2":[0,1,0,0,1]	#document id 2 has two story segments: segment from sentence 1 > 2, segment from sentence 3 > 5	,	"document_id_3":[0,0,1,0,0,1,1]	#document id 1 has three story segments: segment from sentence 1 > 3, segment from sentence 4 > 6	#and segment containing sentence 7	} -------------------------------------------------- | VOCAB | -------------------------------------------------- Another input used in experiments is the vocabulary of the training set. A vocab is a dict stored in a pickle object (*.pckl) The dict() has as keys the words and the value is a unique id to that word. Word ids should start from 1. A special word "<bnd>" is added to the dict and has the id: len(vocab) + 1. This <bnd> word is reserved for unkown words. Words should be lowercased. EXAMPLE: vocab =	{	"word"	: 1,	"usa"	: 2,	"<bnd>" : 3	}