1 changed file
kics.config | ||
Add comment 1 Plus #
Add comment 2 Plus # Author: Hari Sekhon
Add comment 3 Plus # Date: 22/8/2014
Add comment 4 Plus #
Add comment 5 Plus
Add comment 6 Plus # Yarn library support
Add comment 7 Plus
Add comment 8 Plus # Requires SPARK_HOME to be set
Add comment 9 Plus
Add comment 10 Plus __author__ = "Hari Sekhon"
Add comment 11 Plus __version__ = "0.1"
Add comment 12 Plus
Add comment 13 Plus import glob
Add comment 14 Plus import os
Add comment 15 Plus import sys
Add comment 16 Plus
Add comment 17 Plus # This only runs PySpark in local mode, not Yarn mode
Add comment 18 Plus #
Add comment 19 Plus # See ipython-notebook-spark.py for cluster mode (YARN or Standalone)
Add comment 20 Plus
Add comment 21 Plus spark_home = os.getenv('SPARK_HOME', None)
Add comment 22 Plus if not spark_home:
Add comment 23 Plus raise ValueError('SPARK_HOME environment variable is not set')
Add comment 24 Plus sys.path.insert(0, os.path.join(spark_home, 'python'))
Add comment 25 Plus for lib in glob.glob(os.path.join(spark_home, 'python/lib/py4j-*-src.zip')):
Add comment 26 Plus sys.path.insert(0, lib)
Add comment 27 Plus execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))
Add comment 28 Plus