pirateolz.blogg.se

I have a bunch of 3rd party classes, these classes are autogenerated in java and do not have any hierarchy Var writer = df.write.format("csv").options(options)Īny idea why would the write succeed but the read stuck like this? Val options = Map("credentials" -> credential.base64ServiceAccount, "parentProject" -> credential.projectId, "header" -> "true") This is how I previously wrote the object val path = "gs://mybucket/data.csv" My service account has those permissions, I literally added all permissions I could find for Object storage Storage Admin Val options = Map("credentials" -> credential.base64ServiceAccount, "parentProject" -> credential.projectId) Then I simply read like this val path = "gs://mybucket/data.csv" Hc.set("fs.gs.key.id", credential.keyId) Hc.set("fs.gs.project.id", credential.projectId) Hc.set("fs.AbstractFileSystem.gs.impl", ".fs.gcs.GoogleHadoopFS") Hc.set("fs.gs.impl", ".fs.gcs.GoogleHadoopFileSystem") When looking at the logs I see nothing weird, and in the spark UI the job is just stuckīefore I kick the read job I update the spark config as follows: val hc = I previously created with another Spark job locally. I'm trying to read an object with a spark job locally.