spark profiling

============================================================
Profile of RDD<id=6>
============================================================
         3422798588 function calls (3422798076 primitive calls) in 12176.581 seconds

   Ordered by: internal time, cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
1703581040 7212.163    0.000 9940.447    0.000 <stdin>:1(is_valid_english_word)
1703581040 2728.283    0.000 2728.283    0.000 {method 'union' of 'set' objects}
  1756553 2048.382    0.001 11988.828    0.007 <stdin>:1(drop_invalid_tokens)
  1756553  120.803    0.000  120.803    0.000 {method 'split' of 'str' objects}
      128   31.282    0.244 12176.576   95.129 serializers.py:264(dump_stream)
   351464   19.571    0.000   19.571    0.000 {method 'read' of 'file' objects}
   175668    3.950    0.000    3.950    0.000 {cPickle.loads}
  1756553    3.354    0.000    5.319    0.000 <stdin>:1(standardize_tokens)
  1756553    1.846    0.000  122.649    0.000 <stdin>:1(tokenize_document)
  1756553    1.581    0.000    2.102    0.000 <stdin>:9(<lambda>)
  3688774    1.341    0.000    1.341    0.000 {len}
  1756553    1.231    0.000    1.231    0.000 {range}
   175796    1.068    0.000   25.826    0.000 serializers.py:160(_read_with_length)
   175796    0.601    0.000    1.249    0.000 serializers.py:554(read_int)
   175796    0.568    0.000   26.394    0.000 serializers.py:141(load_stream)
   175668    0.301    0.000    4.251    0.000 serializers.py:433(loads)
   175796    0.250    0.000    0.250    0.000 {_struct.unpack}
  640/128    0.002    0.000    0.003    0.000 rdd.py:2406(pipeline_func)
      128    0.001    0.000 12176.581   95.130 worker.py:167(process)
      128    0.001    0.000    0.001    0.000 serializers.py:222(load_stream)
      640    0.001    0.000    0.001    0.000 rdd.py:317(func)
      128    0.000    0.000    0.000    0.000 serializers.py:225(_load_stream_without_unbatching)
      128    0.000    0.000    0.001    0.000 rdd.py:345(func)
      128    0.000    0.000    0.000    0.000 rdd.py:395(func)
      128    0.000    0.000    0.000    0.000 {built-in method from_iterable}
      128    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
      128    0.000    0.000    0.000    0.000 {iter}