No module named Tensorframes on EMR installation #113

I had the same issue when just use pyspark with automatic package installation.

pyspark --packages databricks:tensorframes:0.2.8-s_2.11

So, at the end I decided to build the jars from source and use the jar as extra library, also, you might need to install pandas before running pyspark

PYTHONPATH=tensorframes-assembly-0.2.8.jar \
pyspark --jars tensorframes-assembly-0.2.8.jar

@nikparmar can you tell me more about your current spark setup?

@thunterdb So it's an Amazon EMR setup where I was installing this.

@taklwu I will try and update you

@nikparmar I am not very familiar with EMR's setup. Did the suggestion above work for you?

Hi @thunterdb ,

I have similar issues. I have Spark running in yarn-cluster mode (nodes running CentOS7), version 2.0.1. I added databricks:tensorframes:0.2.9-s_2.11 to spark.jars.packages and it seemed to load the jar according to Spark UI. But I still get no module named tensorframes error when I try to import tensorframes. I have python version 2.7.5 installed across all nodes. I cannot go the other method that sets PYTHONPATH to every node because as I am running Spark in cluster mode, and in this case I will have to maintain it in every single node along with the assemly jars. Is this a bug? If not where do you see the issues are?

Thanks,
Jialiang