Vfrdtyky

I am following the example from spark-redis connector documentation to create Dataframes from a an existing hash.

The hash built as follows:

127.0.0.1:6379> hset person:1 name John age 30

(integer) 2

127.0.0.1:6379> hset person:2 name Peter age 45

(integer) 2

127.0.0.1:6379> hset person:3 name James age 40

The code to read the hash is as follows -

    SparkSession spark = SparkSession

        .builder()

        .appName("MyApp")

        .master("local[*]")

        .config("spark.redis.host", "localhost")

        .config("spark.redis.port", "6379")

         .getOrCreate();

    Dataset<Row> df = spark.read()

       .format("org.apache.spark.sql.redis")

       .schema(new StructType(new StructField {

            DataTypes.createStructField("id", DataTypes.StringType, true),

            DataTypes.createStructField("name", DataTypes.StringType, false),

            DataTypes.createStructField("age", DataTypes.IntegerType, false)

       })

    )

    .option("keys.pattern", "person:*")

    .option("key.column", "id")

    .load();



    df.show();

    df.printSchema();     

Output

+----+-----+---+

|  id| name|age|

+----+-----+---+

|null| John| 30|

|null|James| 40|

|null|Peter| 45|

+----+-----+---+



root

    |-- id: string (nullable = true)

    |-- name: string (nullable = false)

    |-- age: integer (nullable = false)

I was expecting 1, 2, 3 respectively in the Id column, but get null instead. Any pointers in this regard will help. Also this code is in Java so I am not sure if there an issue with data types.

The version you're using doesn't support it - use the master tip until a new version is released.

xref: https://github.com/RedisLabs/spark-redis/issues/114