Write arbitrary POST data to socket with fixed size buffer
Foreword: the code below is micropython and works on a microcontroller, the question is per se, I guess, platform independent and related to how sockets work, which I'm really new.
In an application I send data from i2c eeprom to a server using a socket.
import usocket
buf = bytearray(64)
#...
socket = usocket.socket(*addrinfo[:3])
socket.connect(addrinfo[-1])
#...
socket.write('{0} /{1} HTTP/1.0rn'.format(method, path))
socket.write('Host: {0}rn'.format(host))
socket.write(b'Content-Length: {0}rn'.format(eeprom.payload_size))
socket.write('rn')
#... fill buf with eeprom data...
socket.write(buf)
This works for small amounts of data.
Problem is data can exceed 64 bytes -- the eeprom is 8kbytes and teoretically I could have the need to send thm all -- and I would like to write all data to the same socket reusing the same buffer over and over.
I tried with an iterator, only the first buffer is sent, following socket.write are lost
def ipayload(self, buf):
size = len(buf)
for memaddr in range(self.__head_pointer, self.__tail_pointer, size):
self.read_into(memaddr, buf)
yield min(self.__tail_pointer - memaddr, size)
#...
for c in eeprom.ipayload(buf):
socket.write(buf)
Server side I use PHP and read the whole input stream (I guess):
$content_lenght = $_SERVER['CONTENT_LENGTH'];
$input = fopen('php://input', 'rb');
while (!feof($input) && $data = fread($input, 8)){
$content_lenght -= 8
//use data
}
Also a check on $_SERVER['CONTENT_LENGTH'] and $content_lenght shows I always get 64 bytes, so the first buffer only.
How can I do? Currently I let python allocate memory for a new bytestring every time I read from eeprom, risking I overflow in case it's the full 4Kb to be sent.
python sockets buffer micropython
asked Nov 15 '18 at 9:34
neurinoneurino
6,76922852
|
show 1 more comment
Foreword: the code below is micropython and works on a microcontroller, the question is per se, I guess, platform independent and related to how sockets work, which I'm really new.
In an application I send data from i2c eeprom to a server using a socket.
import usocket
buf = bytearray(64)
#...
socket = usocket.socket(*addrinfo[:3])
socket.connect(addrinfo[-1])
#...
socket.write('{0} /{1} HTTP/1.0rn'.format(method, path))
socket.write('Host: {0}rn'.format(host))
socket.write(b'Content-Length: {0}rn'.format(eeprom.payload_size))
socket.write('rn')
#... fill buf with eeprom data...
socket.write(buf)
This works for small amounts of data.
Problem is data can exceed 64 bytes -- the eeprom is 8kbytes and teoretically I could have the need to send thm all -- and I would like to write all data to the same socket reusing the same buffer over and over.
I tried with an iterator, only the first buffer is sent, following socket.write are lost
def ipayload(self, buf):
size = len(buf)
for memaddr in range(self.__head_pointer, self.__tail_pointer, size):
self.read_into(memaddr, buf)
yield min(self.__tail_pointer - memaddr, size)
#...
for c in eeprom.ipayload(buf):
socket.write(buf)
Server side I use PHP and read the whole input stream (I guess):
$content_lenght = $_SERVER['CONTENT_LENGTH'];
$input = fopen('php://input', 'rb');
while (!feof($input) && $data = fread($input, 8)){
$content_lenght -= 8
//use data
}
Also a check on $_SERVER['CONTENT_LENGTH'] and $content_lenght shows I always get 64 bytes, so the first buffer only.
How can I do? Currently I let python allocate memory for a new bytestring every time I read from eeprom, risking I overflow in case it's the full 4Kb to be sent.
python sockets buffer micropython
asked Nov 15 '18 at 9:34
neurinoneurino
6,76922852
I'm not sure what your are trying to do here. It looks like you are trying to send a HTTP request with a body. But your request is missing thecontent-lengthheader to tell the server up-front how many body data to expect. Maybe this will result in the "following socket.write are lost" you see (you don't explain how exactly you check for this loss).
– Steffen Ullrich
Nov 15 '18 at 10:13
Sorry, I omitted the content-length, I will add it along with server side code
– neurino
Nov 15 '18 at 10:48
@SteffenUllrich added, I hope it gives a better understanding
– neurino
Nov 15 '18 at 10:55
"... a check on$_SERVER['CONTENT_LENGTH']and$content_lenghtshows I always get 64 bytes" - If I understand this correctly your server explicitly sends aContent-length: 64, i.e. theeeprom.payload_sizedoes not specify the size of the full payload you actually want to sent. Of course you need to send the correct size of the full payload.
– Steffen Ullrich
Nov 15 '18 at 11:03
@SteffenUllrich I'm sure because I log it that the microcontroller sendssocket.write(b'Content-Length: 152rn'in the header and the server gets152in$_SERVER['CONTENT_LENGTH']however thewhile (!feof($input) && $data = fread($input, 8))PHP loop exits after 64 bytes read. Any clue?
– neurino
Nov 15 '18 at 11:43
|
show 1 more comment
Foreword: the code below is micropython and works on a microcontroller, the question is per se, I guess, platform independent and related to how sockets work, which I'm really new.
In an application I send data from i2c eeprom to a server using a socket.
import usocket
buf = bytearray(64)
#...
socket = usocket.socket(*addrinfo[:3])
socket.connect(addrinfo[-1])
#...
socket.write('{0} /{1} HTTP/1.0rn'.format(method, path))
socket.write('Host: {0}rn'.format(host))
socket.write(b'Content-Length: {0}rn'.format(eeprom.payload_size))
socket.write('rn')
#... fill buf with eeprom data...
socket.write(buf)
This works for small amounts of data.
Problem is data can exceed 64 bytes -- the eeprom is 8kbytes and teoretically I could have the need to send thm all -- and I would like to write all data to the same socket reusing the same buffer over and over.
I tried with an iterator, only the first buffer is sent, following socket.write are lost
def ipayload(self, buf):
size = len(buf)
for memaddr in range(self.__head_pointer, self.__tail_pointer, size):
self.read_into(memaddr, buf)
yield min(self.__tail_pointer - memaddr, size)
#...
for c in eeprom.ipayload(buf):
socket.write(buf)
Server side I use PHP and read the whole input stream (I guess):
$content_lenght = $_SERVER['CONTENT_LENGTH'];
$input = fopen('php://input', 'rb');
while (!feof($input) && $data = fread($input, 8)){
$content_lenght -= 8
//use data
}
Also a check on $_SERVER['CONTENT_LENGTH'] and $content_lenght shows I always get 64 bytes, so the first buffer only.
How can I do? Currently I let python allocate memory for a new bytestring every time I read from eeprom, risking I overflow in case it's the full 4Kb to be sent.
python sockets buffer micropython
asked Nov 15 '18 at 9:34
neurinoneurino
6,76922852
Foreword: the code below is micropython and works on a microcontroller, the question is per se, I guess, platform independent and related to how sockets work, which I'm really new.
In an application I send data from i2c eeprom to a server using a socket.
import usocket
buf = bytearray(64)
#...
socket = usocket.socket(*addrinfo[:3])
socket.connect(addrinfo[-1])
#...
socket.write('{0} /{1} HTTP/1.0rn'.format(method, path))
socket.write('Host: {0}rn'.format(host))
socket.write(b'Content-Length: {0}rn'.format(eeprom.payload_size))
socket.write('rn')
#... fill buf with eeprom data...
socket.write(buf)
This works for small amounts of data.
Problem is data can exceed 64 bytes -- the eeprom is 8kbytes and teoretically I could have the need to send thm all -- and I would like to write all data to the same socket reusing the same buffer over and over.
I tried with an iterator, only the first buffer is sent, following socket.write are lost
def ipayload(self, buf):
size = len(buf)
for memaddr in range(self.__head_pointer, self.__tail_pointer, size):
self.read_into(memaddr, buf)
yield min(self.__tail_pointer - memaddr, size)
#...
for c in eeprom.ipayload(buf):
socket.write(buf)
Server side I use PHP and read the whole input stream (I guess):
$content_lenght = $_SERVER['CONTENT_LENGTH'];
$input = fopen('php://input', 'rb');
while (!feof($input) && $data = fread($input, 8)){
$content_lenght -= 8
//use data
}
Also a check on $_SERVER['CONTENT_LENGTH'] and $content_lenght shows I always get 64 bytes, so the first buffer only.
How can I do? Currently I let python allocate memory for a new bytestring every time I read from eeprom, risking I overflow in case it's the full 4Kb to be sent.
python sockets buffer micropython
python sockets buffer micropython
asked Nov 15 '18 at 9:34
neurinoneurino
6,76922852
asked Nov 15 '18 at 9:34
neurinoneurino
6,76922852
edited Nov 15 '18 at 10:49
neurino
asked Nov 15 '18 at 9:34
neurinoneurino
6,76922852
asked Nov 15 '18 at 9:34
neurinoneurino
6,76922852
asked Nov 15 '18 at 9:34
neurinoneurino
6,76922852
6,76922852
I'm not sure what your are trying to do here. It looks like you are trying to send a HTTP request with a body. But your request is missing thecontent-lengthheader to tell the server up-front how many body data to expect. Maybe this will result in the "following socket.write are lost" you see (you don't explain how exactly you check for this loss).
– Steffen Ullrich
Nov 15 '18 at 10:13
Sorry, I omitted the content-length, I will add it along with server side code
– neurino
Nov 15 '18 at 10:48
@SteffenUllrich added, I hope it gives a better understanding
– neurino
Nov 15 '18 at 10:55
"... a check on$_SERVER['CONTENT_LENGTH']and$content_lenghtshows I always get 64 bytes" - If I understand this correctly your server explicitly sends aContent-length: 64, i.e. theeeprom.payload_sizedoes not specify the size of the full payload you actually want to sent. Of course you need to send the correct size of the full payload.
– Steffen Ullrich
Nov 15 '18 at 11:03
@SteffenUllrich I'm sure because I log it that the microcontroller sendssocket.write(b'Content-Length: 152rn'in the header and the server gets152in$_SERVER['CONTENT_LENGTH']however thewhile (!feof($input) && $data = fread($input, 8))PHP loop exits after 64 bytes read. Any clue?
– neurino
Nov 15 '18 at 11:43
|
show 1 more comment
I'm not sure what your are trying to do here. It looks like you are trying to send a HTTP request with a body. But your request is missing thecontent-lengthheader to tell the server up-front how many body data to expect. Maybe this will result in the "following socket.write are lost" you see (you don't explain how exactly you check for this loss).
– Steffen Ullrich
Nov 15 '18 at 10:13
Sorry, I omitted the content-length, I will add it along with server side code
– neurino
Nov 15 '18 at 10:48
@SteffenUllrich added, I hope it gives a better understanding
– neurino
Nov 15 '18 at 10:55
"... a check on$_SERVER['CONTENT_LENGTH']and$content_lenghtshows I always get 64 bytes" - If I understand this correctly your server explicitly sends aContent-length: 64, i.e. theeeprom.payload_sizedoes not specify the size of the full payload you actually want to sent. Of course you need to send the correct size of the full payload.
– Steffen Ullrich
Nov 15 '18 at 11:03
@SteffenUllrich I'm sure because I log it that the microcontroller sendssocket.write(b'Content-Length: 152rn'in the header and the server gets152in$_SERVER['CONTENT_LENGTH']however thewhile (!feof($input) && $data = fread($input, 8))PHP loop exits after 64 bytes read. Any clue?
– neurino
Nov 15 '18 at 11:43
I'm not sure what your are trying to do here. It looks like you are trying to send a HTTP request with a body. But your request is missing the
content-length header to tell the server up-front how many body data to expect. Maybe this will result in the "following socket.write are lost" you see (you don't explain how exactly you check for this loss).– Steffen Ullrich
Nov 15 '18 at 10:13
I'm not sure what your are trying to do here. It looks like you are trying to send a HTTP request with a body. But your request is missing the
content-length header to tell the server up-front how many body data to expect. Maybe this will result in the "following socket.write are lost" you see (you don't explain how exactly you check for this loss).– Steffen Ullrich
Nov 15 '18 at 10:13
Sorry, I omitted the content-length, I will add it along with server side code
– neurino
Nov 15 '18 at 10:48
Sorry, I omitted the content-length, I will add it along with server side code
– neurino
Nov 15 '18 at 10:48
@SteffenUllrich added, I hope it gives a better understanding
– neurino
Nov 15 '18 at 10:55
@SteffenUllrich added, I hope it gives a better understanding
– neurino
Nov 15 '18 at 10:55
"... a check on
$_SERVER['CONTENT_LENGTH'] and $content_lenght shows I always get 64 bytes" - If I understand this correctly your server explicitly sends a Content-length: 64, i.e. the eeprom.payload_size does not specify the size of the full payload you actually want to sent. Of course you need to send the correct size of the full payload.– Steffen Ullrich
Nov 15 '18 at 11:03
"... a check on
$_SERVER['CONTENT_LENGTH'] and $content_lenght shows I always get 64 bytes" - If I understand this correctly your server explicitly sends a Content-length: 64, i.e. the eeprom.payload_size does not specify the size of the full payload you actually want to sent. Of course you need to send the correct size of the full payload.– Steffen Ullrich
Nov 15 '18 at 11:03
@SteffenUllrich I'm sure because I log it that the microcontroller sends
socket.write(b'Content-Length: 152rn' in the header and the server gets 152 in $_SERVER['CONTENT_LENGTH'] however the while (!feof($input) && $data = fread($input, 8)) PHP loop exits after 64 bytes read. Any clue?– neurino
Nov 15 '18 at 11:43
@SteffenUllrich I'm sure because I log it that the microcontroller sends
socket.write(b'Content-Length: 152rn' in the header and the server gets 152 in $_SERVER['CONTENT_LENGTH'] however the while (!feof($input) && $data = fread($input, 8)) PHP loop exits after 64 bytes read. Any clue?– neurino
Nov 15 '18 at 11:43
|
show 1 more comment
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53316320%2fwrite-arbitrary-post-data-to-socket-with-fixed-size-buffer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53316320%2fwrite-arbitrary-post-data-to-socket-with-fixed-size-buffer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I'm not sure what your are trying to do here. It looks like you are trying to send a HTTP request with a body. But your request is missing the
content-lengthheader to tell the server up-front how many body data to expect. Maybe this will result in the "following socket.write are lost" you see (you don't explain how exactly you check for this loss).– Steffen Ullrich
Nov 15 '18 at 10:13
Sorry, I omitted the content-length, I will add it along with server side code
– neurino
Nov 15 '18 at 10:48
@SteffenUllrich added, I hope it gives a better understanding
– neurino
Nov 15 '18 at 10:55
"... a check on
$_SERVER['CONTENT_LENGTH']and$content_lenghtshows I always get 64 bytes" - If I understand this correctly your server explicitly sends aContent-length: 64, i.e. theeeprom.payload_sizedoes not specify the size of the full payload you actually want to sent. Of course you need to send the correct size of the full payload.– Steffen Ullrich
Nov 15 '18 at 11:03
@SteffenUllrich I'm sure because I log it that the microcontroller sends
socket.write(b'Content-Length: 152rn'in the header and the server gets152in$_SERVER['CONTENT_LENGTH']however thewhile (!feof($input) && $data = fread($input, 8))PHP loop exits after 64 bytes read. Any clue?– neurino
Nov 15 '18 at 11:43