is there a way convert a spark dataframe generated from a sql statement into an rdd?











up vote
0
down vote

favorite












if i use this spark sql statement:



df = spark.sql('SELECT col_name FROM table_name')


it will return a spark dataframe object. How can i convert this to an rdd? is there a way to read a table directly using sql but generating an rdd instead of a dataframe?



Thanks in advance






python python-3.x apache-spark dataframe pyspark







share|improve this question


















  • 1




    df.rdd should give you the RDD
    – sramalingam24
    Nov 11 at 16:28










  • i tried that but no, instead, i get the following error: PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79)
    – Miguel 2488
    Nov 11 at 16:32










  • i have visited a good bunch of posts here talking about more or less the same thing, but i get this error instead
    – Miguel 2488
    Nov 11 at 16:33






  • 1




    stackoverflow.com/questions/29000514/… Try one of the alternatives suggested here
    – sramalingam24
    Nov 11 at 16:40










  • Possible duplicate of Pyspark - Why i can't convert a sql dataframe to an rdd?
    – user10465355
    Nov 11 at 16:40















up vote
0
down vote

favorite












if i use this spark sql statement:



df = spark.sql('SELECT col_name FROM table_name')


it will return a spark dataframe object. How can i convert this to an rdd? is there a way to read a table directly using sql but generating an rdd instead of a dataframe?



Thanks in advance






python python-3.x apache-spark dataframe pyspark







share|improve this question


















  • 1




    df.rdd should give you the RDD
    – sramalingam24
    Nov 11 at 16:28










  • i tried that but no, instead, i get the following error: PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79)
    – Miguel 2488
    Nov 11 at 16:32










  • i have visited a good bunch of posts here talking about more or less the same thing, but i get this error instead
    – Miguel 2488
    Nov 11 at 16:33






  • 1




    stackoverflow.com/questions/29000514/… Try one of the alternatives suggested here
    – sramalingam24
    Nov 11 at 16:40










  • Possible duplicate of Pyspark - Why i can't convert a sql dataframe to an rdd?
    – user10465355
    Nov 11 at 16:40













up vote
0
down vote

favorite









up vote
0
down vote

favorite











if i use this spark sql statement:



df = spark.sql('SELECT col_name FROM table_name')


it will return a spark dataframe object. How can i convert this to an rdd? is there a way to read a table directly using sql but generating an rdd instead of a dataframe?



Thanks in advance






python python-3.x apache-spark dataframe pyspark







share|improve this question













if i use this spark sql statement:



df = spark.sql('SELECT col_name FROM table_name')


it will return a spark dataframe object. How can i convert this to an rdd? is there a way to read a table directly using sql but generating an rdd instead of a dataframe?



Thanks in advance






python python-3.x apache-spark dataframe pyspark




python python-3.x apache-spark dataframe pyspark






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 11 at 16:26









Miguel 2488

18712




18712








  • 1




    df.rdd should give you the RDD
    – sramalingam24
    Nov 11 at 16:28










  • i tried that but no, instead, i get the following error: PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79)
    – Miguel 2488
    Nov 11 at 16:32










  • i have visited a good bunch of posts here talking about more or less the same thing, but i get this error instead
    – Miguel 2488
    Nov 11 at 16:33






  • 1




    stackoverflow.com/questions/29000514/… Try one of the alternatives suggested here
    – sramalingam24
    Nov 11 at 16:40










  • Possible duplicate of Pyspark - Why i can't convert a sql dataframe to an rdd?
    – user10465355
    Nov 11 at 16:40














  • 1




    df.rdd should give you the RDD
    – sramalingam24
    Nov 11 at 16:28










  • i tried that but no, instead, i get the following error: PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79)
    – Miguel 2488
    Nov 11 at 16:32










  • i have visited a good bunch of posts here talking about more or less the same thing, but i get this error instead
    – Miguel 2488
    Nov 11 at 16:33






  • 1




    stackoverflow.com/questions/29000514/… Try one of the alternatives suggested here
    – sramalingam24
    Nov 11 at 16:40










  • Possible duplicate of Pyspark - Why i can't convert a sql dataframe to an rdd?
    – user10465355
    Nov 11 at 16:40








1




1




df.rdd should give you the RDD
– sramalingam24
Nov 11 at 16:28




df.rdd should give you the RDD
– sramalingam24
Nov 11 at 16:28












i tried that but no, instead, i get the following error: PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79)
– Miguel 2488
Nov 11 at 16:32




i tried that but no, instead, i get the following error: PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79)
– Miguel 2488
Nov 11 at 16:32












i have visited a good bunch of posts here talking about more or less the same thing, but i get this error instead
– Miguel 2488
Nov 11 at 16:33




i have visited a good bunch of posts here talking about more or less the same thing, but i get this error instead
– Miguel 2488
Nov 11 at 16:33




1




1




stackoverflow.com/questions/29000514/… Try one of the alternatives suggested here
– sramalingam24
Nov 11 at 16:40




stackoverflow.com/questions/29000514/… Try one of the alternatives suggested here
– sramalingam24
Nov 11 at 16:40












Possible duplicate of Pyspark - Why i can't convert a sql dataframe to an rdd?
– user10465355
Nov 11 at 16:40




Possible duplicate of Pyspark - Why i can't convert a sql dataframe to an rdd?
– user10465355
Nov 11 at 16:40












1 Answer
1






active

oldest

votes

















up vote
2
down vote



accepted










df = spark.sql('SELECT col_name FROM table_name')


df.rdd # you can save it, perform transformations etc.



df.rdd returns the content as an pyspark.RDD of Row.



You can then map on that RDD of Row transforming every Row into a numpy vector. I can't be more specific about the transformation since I don't know what your vector represents with the information given.



Note 1: df is the variable define our Dataframe.



Note 2: this function is available since Spark 1.3






share|improve this answer























  • Did this solve the PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) error?
    – karma4917
    Nov 12 at 16:13











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53250760%2fis-there-a-way-convert-a-spark-dataframe-generated-from-a-sql-statement-into-an%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
2
down vote



accepted










df = spark.sql('SELECT col_name FROM table_name')


df.rdd # you can save it, perform transformations etc.



df.rdd returns the content as an pyspark.RDD of Row.



You can then map on that RDD of Row transforming every Row into a numpy vector. I can't be more specific about the transformation since I don't know what your vector represents with the information given.



Note 1: df is the variable define our Dataframe.



Note 2: this function is available since Spark 1.3






share|improve this answer























  • Did this solve the PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) error?
    – karma4917
    Nov 12 at 16:13















up vote
2
down vote



accepted










df = spark.sql('SELECT col_name FROM table_name')


df.rdd # you can save it, perform transformations etc.



df.rdd returns the content as an pyspark.RDD of Row.



You can then map on that RDD of Row transforming every Row into a numpy vector. I can't be more specific about the transformation since I don't know what your vector represents with the information given.



Note 1: df is the variable define our Dataframe.



Note 2: this function is available since Spark 1.3






share|improve this answer























  • Did this solve the PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) error?
    – karma4917
    Nov 12 at 16:13













up vote
2
down vote



accepted







up vote
2
down vote



accepted






df = spark.sql('SELECT col_name FROM table_name')


df.rdd # you can save it, perform transformations etc.



df.rdd returns the content as an pyspark.RDD of Row.



You can then map on that RDD of Row transforming every Row into a numpy vector. I can't be more specific about the transformation since I don't know what your vector represents with the information given.



Note 1: df is the variable define our Dataframe.



Note 2: this function is available since Spark 1.3






share|improve this answer














df = spark.sql('SELECT col_name FROM table_name')


df.rdd # you can save it, perform transformations etc.



df.rdd returns the content as an pyspark.RDD of Row.



You can then map on that RDD of Row transforming every Row into a numpy vector. I can't be more specific about the transformation since I don't know what your vector represents with the information given.



Note 1: df is the variable define our Dataframe.



Note 2: this function is available since Spark 1.3







share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 12 at 7:38









Ali AzG

587515




587515










answered Nov 11 at 20:29









Nagilla Venkatesh

362




362












  • Did this solve the PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) error?
    – karma4917
    Nov 12 at 16:13


















  • Did this solve the PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) error?
    – karma4917
    Nov 12 at 16:13
















Did this solve the PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) error?
– karma4917
Nov 12 at 16:13




Did this solve the PicklingError: Could not serialize object: Py4JError: An error occurred while calling o60.__getstate__. Trace: py4j.Py4JException: Method __getstate__() does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) error?
– karma4917
Nov 12 at 16:13


















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53250760%2fis-there-a-way-convert-a-spark-dataframe-generated-from-a-sql-statement-into-an%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

List item for chat from Array inside array React Native

Image
0 This is my code for Chat Box, the window where I have "In" and "Out" messages are appearing. import React, { Component } from "react"; import { StyleSheet, Text, View, TouchableOpacity, TextInput, FlatList, Platform, AsyncStorage } from "react-native"; import Tutor from "../image/krutika.jpg"; import { Container, Header, Left, Input, Body, Right, Thumbnail, Button } from "native-base"; import FontAwesome from "react-native-vector-icons/FontAwesome"; import Ionicons from "react-native-vector-icons/Ionicons"; import Icon1 from "react-native-vector-icons/FontAwesome"; import axios from "axios"; export default class ChatBox extends Component { st...
Read more

Thiostrepton

Image
Thiostrepton [1] Names Other names Alaninamide, Bryamycin, Thiactin Identifiers CAS Number 1393-48-2   Y 3D model (JSmol) Interactive image ChEMBL ChEMBL468719   N ECHA InfoCard 100.014.304 PubChem CID 16130278 InChI InChI=1S/C72H85N19O18S5/c1-14-26(3)47-63(105)78-30(7)57(99)75-28(5)56(98)76-31(8)58(100)91-72-19-18-40(66-85-43(22-111-66)59(101)77-29(6)55(97)74-27(4)54(73)96)81-52(72)42-21-112-67(83-42)49(34(11)109-69(107)41-20-37(32(9)92)36-16-17-39(79-47)51(95)50(36)80-41)89-60(102)44-24-113-68(86-44)53(71(13,108)35(12)94)90-62(104)45-23-110-65(84-45)38(15-2)82-64(106)48(33(10)93)88-61(103)46-25-114-70(72)87-46/h15-17,20-22,24-26,30-35,39,45,47-49,51-53,79,92-95,108H,4-6,14,18-19,23H2,1-3,7-13H3,(H2,73,96)(H,74,97)(H,75,99)(H,76,98)(H,77,101)(H,78,105)(H,82,106)(H,88,103)(H,89,102)(H,90,104)(H,91,100)/b38-15-/t26-,30-,31-,32-,33+,34+,35+,39+,45+,47-,48-,49-,51-,52+,53+,71+,72+/m0/s1 Key: NSFFHOGKXHRQEW-AIHSUZKVSA-N ...
Read more

Caerphilly

Image
This article is about the town of Caerphilly. For the cheese of the same name, see Caerphilly cheese. For other uses, see Caerphilly (disambiguation). Caerphilly Welsh: Caerffili Caerphilly Castle Caerphilly Location within Caerphilly Population 30,388  OS grid reference ST1586 Principal area Caerphilly Ceremonial county Gwent Country Wales Sovereign state United Kingdom Post town CAERPHILLY Postcode district CF83 Dialling code 029 Police Gwent Fire South Wales Ambulance Welsh EU Parliament Wales UK Parliament Caerphilly Welsh Assembly Caerphilly List of places UK Wales Caerphilly 51°34′41″N 3°13′05″W  /  51.578°N 3.218°W  / 51.578; -3.218 Coordinates: 51°34′41″N 3°13′05″W  /  51.578°N 3.218°W  / 51.578; -3.218 Caerphilly ( / k aɪər ˈ f ɪ l i / ; Welsh: Caerffili , Welsh pronunciation:  [ˌkɑːɨrˈfɪlɪ] ) is a town and community in South Wales, at the southern end of the Rhymney Valley....
Read more