human
3.  HW5.3.2  Full-scale experiment: EDA of Google n-grams dataset (PHASE 2)
Back to Table of Contents
Do some EDA on this dataset using mrjob, e.g., 

Longest 5-gram (number of characters)
Top 10 most frequent words (please use the count information), i.e., unigrams
20 Most/Least densely appearing words (count/pages_count) sorted in decreasing order of relative frequency 
Distribution of 5-gram sizes (character length).  E.g., count (using the count field) up how many times a 5-gram of 50 characters shows up. Plot the data graphically using a histogram.

We included all of this analysis at the end.
See here.
<a name="2.2">
2.2 Stripe Creation
Back to Table of Contents  
The section takes the output from 2.1 and create stripes based on the vocabularies identified.
 What is a the Python code you requested?
assistant
#Here is the Python code you requested.
%%writefile MakeStripes.py

from mrjob.job import MRJob
from collections import Counter
from sys import stderr
from re import findall

class MakeStripes(MRJob):
    def mapper_init(self):
        """
        Read in index words and word list.
        """
        self.stripes = {}
        
        self.indexlist, self.wordslist = [],[]
        with open('vocabs', 'r') as vocabFile:
            for line in vocabFile:
                word_type, word = line.replace('"', '').split()
                if word_type == 'index':
                    self.indexlist.append(word)
                else:
                    self.wordslist.append(word)
        
        # Convert to sets to make lookups faster
        self.indexlist = set(self.indexlist)
        self.wordslist = set(self.wordslist)
    
    def mapper(self, _, lines):
        """
        Make stripes using index and words list
        """
        terms, term_count, page_count, book_count = lines.split("\t")
        term_count = int(term_count)
        terms = findall(r'[a-z]+', terms.lower())
                
        for item in terms:
            if item in self.indexlist:
                for val in terms:
                    if val != item and val in self.wordslist:
                        yield item, {val:term_count}
        
    def combiner(self, keys, values):
        values_sum = Counter()
        for val in values:
            values_sum += Counter(val)
        yield keys, dict(values_sum)

    def reducer(self, keys, values):
        values_sum = Counter()
        for val in values:
            values_sum += Counter(val)
        yield keys, dict(values_sum)
        
if __name__ == "__main__":
    MakeStripes.run()

!python MakeStripes.py --file vocabs mini_5gram.txt -q