Vfrdtyky

SPARK-26039

While loading empty orc folder. Anyways to bypass this.

val df = spark.read.format("orc").load(orcFolderPath)



org.apache.spark.sql.AnalysisException: Unable to infer schema for ORC. It must be specified manually.;

  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:185)

  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:185)

  at scala.Option.getOrElse(Option.scala:121)

  at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:184)

  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)

  at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)

  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)

  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)

  ... 49 elided

Getting this error may be orc reader trying to infer schema but i want to bypass this special case when somehow in repository blank folder came up but has to be checked.

try {

    spark.read.format("orc").load(path)

    } catch {

        case ex: org.apache.spark.sql.AnalysisException => {

        null

            }

    }

Tried by this way to catch exception. Any other way would be helpful

got one more solution seems like...this is also not best one...

import org.apache.hadoop.conf.Configuration

import org.apache.hadoop.fs.{FileSystem, Path}



  def pathStatus(path: String): Boolean = {

      val config: Configuration = new Configuration()

      val fs: FileSystem = FileSystem.get(config)

    if (fs.globStatus(new Path(path)) == null) {

      false

    } else {

      true

    }

  }