Struct unpack on win32file.DeviceIoControl












0















I am trying to understand and work with win32file. I need to grab USN Journals and having a hard time understanding code snippets I found online. This is the code snippet I found -



format = 'qqqqqLLLLqqqqq'
length = struct.calcsize(format)
out_buffer = win32file.DeviceIoControl(volh, winioctlcon.FSCTL_GET_NTFS_VOLUME_DATA, None, length)
data = struct.unpack(format, out_buffer)


Now I am really rusty when it comes to C and it's structures. What I have understood for now is format is 96 bytes buffer and it'll get the output from DeviceIoControl



So I tried to change the format to 'QQQQQQQQQQQQQQQQQQQ' to see what happens(to see because I am kind of clueless what might actually happen) and it turns out I got a larger out_buffer this time. So I thought to unpack it -



struct.unpack(format, out_buffer)


And surprise to me, I got -



struct.error: unpack requires a string argument of length 152


So I added another 'Q' to increase the size and got the same result. I don't understand why 'qqqqqLLLLqqqqq' works and 'QQQQQQQQQQQQQQQQQQQ' does not. So my questions are -




  • My understanding was we can unpack if buffer was larger than the output so why doesn't the unpack work?


  • Would I have to remember these formats every-time I want to get something out from DeviceIoControl?



Pointing me out to resources would also be an added bonus as I need to build on the code to read USN Journals and I don't think hit-and-try is going to get me anywhere










share|improve this question

























  • What does volh's creation process looks like (CreateFile call)?

    – CristiFati
    Nov 16 '18 at 23:49
















0















I am trying to understand and work with win32file. I need to grab USN Journals and having a hard time understanding code snippets I found online. This is the code snippet I found -



format = 'qqqqqLLLLqqqqq'
length = struct.calcsize(format)
out_buffer = win32file.DeviceIoControl(volh, winioctlcon.FSCTL_GET_NTFS_VOLUME_DATA, None, length)
data = struct.unpack(format, out_buffer)


Now I am really rusty when it comes to C and it's structures. What I have understood for now is format is 96 bytes buffer and it'll get the output from DeviceIoControl



So I tried to change the format to 'QQQQQQQQQQQQQQQQQQQ' to see what happens(to see because I am kind of clueless what might actually happen) and it turns out I got a larger out_buffer this time. So I thought to unpack it -



struct.unpack(format, out_buffer)


And surprise to me, I got -



struct.error: unpack requires a string argument of length 152


So I added another 'Q' to increase the size and got the same result. I don't understand why 'qqqqqLLLLqqqqq' works and 'QQQQQQQQQQQQQQQQQQQ' does not. So my questions are -




  • My understanding was we can unpack if buffer was larger than the output so why doesn't the unpack work?


  • Would I have to remember these formats every-time I want to get something out from DeviceIoControl?



Pointing me out to resources would also be an added bonus as I need to build on the code to read USN Journals and I don't think hit-and-try is going to get me anywhere










share|improve this question

























  • What does volh's creation process looks like (CreateFile call)?

    – CristiFati
    Nov 16 '18 at 23:49














0












0








0








I am trying to understand and work with win32file. I need to grab USN Journals and having a hard time understanding code snippets I found online. This is the code snippet I found -



format = 'qqqqqLLLLqqqqq'
length = struct.calcsize(format)
out_buffer = win32file.DeviceIoControl(volh, winioctlcon.FSCTL_GET_NTFS_VOLUME_DATA, None, length)
data = struct.unpack(format, out_buffer)


Now I am really rusty when it comes to C and it's structures. What I have understood for now is format is 96 bytes buffer and it'll get the output from DeviceIoControl



So I tried to change the format to 'QQQQQQQQQQQQQQQQQQQ' to see what happens(to see because I am kind of clueless what might actually happen) and it turns out I got a larger out_buffer this time. So I thought to unpack it -



struct.unpack(format, out_buffer)


And surprise to me, I got -



struct.error: unpack requires a string argument of length 152


So I added another 'Q' to increase the size and got the same result. I don't understand why 'qqqqqLLLLqqqqq' works and 'QQQQQQQQQQQQQQQQQQQ' does not. So my questions are -




  • My understanding was we can unpack if buffer was larger than the output so why doesn't the unpack work?


  • Would I have to remember these formats every-time I want to get something out from DeviceIoControl?



Pointing me out to resources would also be an added bonus as I need to build on the code to read USN Journals and I don't think hit-and-try is going to get me anywhere










share|improve this question
















I am trying to understand and work with win32file. I need to grab USN Journals and having a hard time understanding code snippets I found online. This is the code snippet I found -



format = 'qqqqqLLLLqqqqq'
length = struct.calcsize(format)
out_buffer = win32file.DeviceIoControl(volh, winioctlcon.FSCTL_GET_NTFS_VOLUME_DATA, None, length)
data = struct.unpack(format, out_buffer)


Now I am really rusty when it comes to C and it's structures. What I have understood for now is format is 96 bytes buffer and it'll get the output from DeviceIoControl



So I tried to change the format to 'QQQQQQQQQQQQQQQQQQQ' to see what happens(to see because I am kind of clueless what might actually happen) and it turns out I got a larger out_buffer this time. So I thought to unpack it -



struct.unpack(format, out_buffer)


And surprise to me, I got -



struct.error: unpack requires a string argument of length 152


So I added another 'Q' to increase the size and got the same result. I don't understand why 'qqqqqLLLLqqqqq' works and 'QQQQQQQQQQQQQQQQQQQ' does not. So my questions are -




  • My understanding was we can unpack if buffer was larger than the output so why doesn't the unpack work?


  • Would I have to remember these formats every-time I want to get something out from DeviceIoControl?



Pointing me out to resources would also be an added bonus as I need to build on the code to read USN Journals and I don't think hit-and-try is going to get me anywhere







python struct pywin32 deviceiocontrol






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 15 '18 at 12:17







ThatBird

















asked Nov 15 '18 at 11:53









ThatBirdThatBird

1,8961619




1,8961619













  • What does volh's creation process looks like (CreateFile call)?

    – CristiFati
    Nov 16 '18 at 23:49



















  • What does volh's creation process looks like (CreateFile call)?

    – CristiFati
    Nov 16 '18 at 23:49

















What does volh's creation process looks like (CreateFile call)?

– CristiFati
Nov 16 '18 at 23:49





What does volh's creation process looks like (CreateFile call)?

– CristiFati
Nov 16 '18 at 23:49












1 Answer
1






active

oldest

votes


















2














Let's split the problem in smaller pieces and take each one at a time.





  • win32file module is part of [GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions which is a Python wrapper over WinAPIs




    • Unfortunately, it doesn't have an official doc page (or I'm not aware of one), so below is the best that I could find (I've been using this for years). An alternative method that never fails (but it's less appealing) is looking directly at the code


    • [ActiveState]: win32file.DeviceIoControl is a wrapper over [MS.Docs]: DeviceIoControl function





  • DeviceIoControl acts differently, depending on the dwIoControlCode (2nd argument). For FSCTL_GET_NTFS_VOLUME_DATA, it fills a buffer with volume specific data. From [MSDN]: FSCTL_GET_NTFS_VOLUME_DATA control code:




    lpOutBuffer
    A pointer to the output buffer, an NTFS_VOLUME_DATA_BUFFER (@CristiFati: !!! Broken URL !!!) structure. The file record associated with the file identifier specified in the input buffer is returned in this buffer. Refer to the Remarks section of the documentation for the NTFS_VOLUME_DATA_BUFFER structure for specific information on how to determine the correct size of this buffer.




    Here's an alternative to the above broken URL: [MSDN]: NTFS_VOLUME_DATA_BUFFER structure. As I'm not sure for how long will it be valid, I'm pasting the structure definition below (from Windows Kits 8.1: winioctl.h (line #4987)):



    typedef struct {

    LARGE_INTEGER VolumeSerialNumber;
    LARGE_INTEGER NumberSectors;
    LARGE_INTEGER TotalClusters;
    LARGE_INTEGER FreeClusters;
    LARGE_INTEGER TotalReserved;
    DWORD BytesPerSector;
    DWORD BytesPerCluster;
    DWORD BytesPerFileRecordSegment;
    DWORD ClustersPerFileRecordSegment;
    LARGE_INTEGER MftValidDataLength;
    LARGE_INTEGER MftStartLcn;
    LARGE_INTEGER Mft2StartLcn;
    LARGE_INTEGER MftZoneStart;
    LARGE_INTEGER MftZoneEnd;

    } NTFS_VOLUME_DATA_BUFFER, *PNTFS_VOLUME_DATA_BUFFER;


  • [Python 3]: struct - Interpret bytes as packed binary data module, is used for conversions between binary and "normal" data. It contains all the format characters meanings (q, Q, L, ...), and much more. You could also take a look at [SO]: Python struct.pack() behavior for more (practical) details



After going over the above materials, things should become clearer.



A couple of notes:




  • If one doesn't know what a function does (returns), they should probably don't use it (without reading the manual, of course). Although nowadays, both Win (which always had a lot of restrictions for the regular user) and Ux "protect users from themselves" (e.g.: root login no longer allowed, write protect %SystemDrive%)

  • The attempts (trial and error) show some lack of experience (probably every does it at some point, the key is not to rely solely on it)

  • "Would I have to remember these formats every-time I want to get something out from DeviceIoControl"?


    • Again, if not knowing that a function does, what's the reason for calling it? If you meant learning NTFS_VOLUME_DATA_BUFFER by heart, it's definitely not the case. You should know its structure only when using it (and as you've noticed there are some places that you can get it from - including this very post :) )



  • "My understanding was we can unpack if buffer was larger than the output so why doesn't the unpack work?"


    • Your understanding is correct. But win32file.DeviceIoControl seems to sometimes (probably when reaching 1stNULL after 96 bytes) truncate the output buffer when passing a value greater than the expected one (via length argument). When passing a smaller one, it will fail (as expected)




I've also prepared a dummy Python example.



code.py:



#!/usr/bin/env python3

import sys
import struct
import win32file
import win32api
import win32con
import winioctlcon


VOLUME_LETTER = "E"

FILE_READ_ATTRIBUTES = 0x0080
FILE_EXECUTE = 0x0020

vol_data_buf_fmt = "qqqqqLLLLqqqqq" # This is the format that matches NTFS_VOLUME_DATA_BUFFER definition (96 bytes). Note: Instead of each 'q' you could also use 'Ll' as 'LARGE_INTEGER' is an union

BINARY_FORMAT_LIST = [
vol_data_buf_fmt,
"QQQQQQQQQQQQQQQQQQQ",
]


def print_formats(): # Dummy func
print("Formats and lengths:")
for format in BINARY_FORMAT_LIST:
print(" {:s}: {:d}".format(format, struct.calcsize(format)))


def main():
#print_formats()
vol_unc_name = "\\.\{:s}:".format(VOLUME_LETTER)
print("volume: ", vol_unc_name)
access_flags = FILE_READ_ATTRIBUTES | FILE_EXECUTE # Apparently, doesn't work without FILE_EXECUTE
share_flags = win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE # Doesn't work withou FILE_SHARE_WRITE
creation_flags = win32con.OPEN_EXISTING
attributes_flags = win32con.FILE_ATTRIBUTE_NORMAL
vol_handle = win32file.CreateFile(vol_unc_name, access_flags, share_flags, None, creation_flags, attributes_flags, None)

buf_len = struct.calcsize(vol_data_buf_fmt)
for i in [buf_len]:
print(" Passing a buffer size of: {:d}".format(i))
buf = win32file.DeviceIoControl(vol_handle, winioctlcon.FSCTL_GET_NTFS_VOLUME_DATA, None, i)
print(" DeviceIocontrol returned a {:d} bytes long {:}".format(len(buf), type(buf)))
out = struct.unpack_from(vol_data_buf_fmt, buf)
print("n NumberSectors: {:}n TotalClusters: {:}n BytesPerCluster: {:}".format(out[1], out[2], out[6]))
win32api.CloseHandle(vol_handle)


if __name__ == "__main__":
print("Python {:s} on {:s}n".format(sys.version, sys.platform))
main()


Output:




(py35x64_test) e:WorkDevStackOverflowq053318932>"e:WorkDevVEnvspy35x64_testScriptspython.exe" ./code.py
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

volume: \.E:
Passing a buffer size of: 96
DeviceIocontrol returned a 96 bytes long <class 'bytes'>

NumberSectors: 494374911
TotalClusters: 61796863
BytesPerCluster: 4096



Needless to say that multiplying TotalClusters by BytesPerCluster, I get the correct bytes number (as reported by Win) for my E: drive.






share|improve this answer


























  • Thanks @CristiFati. Actually, with a bit of hit and try and research, I was able to find the resources mentioned in your comment and was thinking of answering my own question if nobody answered it but you have

    – ThatBird
    Nov 19 '18 at 10:32











  • "protect users from themselves" as you said so I knew the risks I was getting into while poking around with these commands

    – ThatBird
    Nov 19 '18 at 10:32











  • "Every attempt simply expresses lack of experience" but attempt adds up to the experience, which in turn helped me to discover these docs

    – ThatBird
    Nov 19 '18 at 10:34











  • I can query through the journal now but I need to build path to parent file, this is where I am stuck currently. What I have found out is there are to ways to get this to work, one is going through the MFT(I don't want that) and the other is opening file with File Reference Number. Can you help me with the second one?

    – ThatBird
    Nov 19 '18 at 10:37











  • I'm afraid that my knowledge on "File Reference Number". Is somewhere close to 0. So without some code samples I can't do anything. But, wouldn't that be a different question (cause the code from this one is related to volume properties rather than volume changes)?

    – CristiFati
    Nov 19 '18 at 13:33













Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53318932%2fstruct-unpack-on-win32file-deviceiocontrol%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









2














Let's split the problem in smaller pieces and take each one at a time.





  • win32file module is part of [GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions which is a Python wrapper over WinAPIs




    • Unfortunately, it doesn't have an official doc page (or I'm not aware of one), so below is the best that I could find (I've been using this for years). An alternative method that never fails (but it's less appealing) is looking directly at the code


    • [ActiveState]: win32file.DeviceIoControl is a wrapper over [MS.Docs]: DeviceIoControl function





  • DeviceIoControl acts differently, depending on the dwIoControlCode (2nd argument). For FSCTL_GET_NTFS_VOLUME_DATA, it fills a buffer with volume specific data. From [MSDN]: FSCTL_GET_NTFS_VOLUME_DATA control code:




    lpOutBuffer
    A pointer to the output buffer, an NTFS_VOLUME_DATA_BUFFER (@CristiFati: !!! Broken URL !!!) structure. The file record associated with the file identifier specified in the input buffer is returned in this buffer. Refer to the Remarks section of the documentation for the NTFS_VOLUME_DATA_BUFFER structure for specific information on how to determine the correct size of this buffer.




    Here's an alternative to the above broken URL: [MSDN]: NTFS_VOLUME_DATA_BUFFER structure. As I'm not sure for how long will it be valid, I'm pasting the structure definition below (from Windows Kits 8.1: winioctl.h (line #4987)):



    typedef struct {

    LARGE_INTEGER VolumeSerialNumber;
    LARGE_INTEGER NumberSectors;
    LARGE_INTEGER TotalClusters;
    LARGE_INTEGER FreeClusters;
    LARGE_INTEGER TotalReserved;
    DWORD BytesPerSector;
    DWORD BytesPerCluster;
    DWORD BytesPerFileRecordSegment;
    DWORD ClustersPerFileRecordSegment;
    LARGE_INTEGER MftValidDataLength;
    LARGE_INTEGER MftStartLcn;
    LARGE_INTEGER Mft2StartLcn;
    LARGE_INTEGER MftZoneStart;
    LARGE_INTEGER MftZoneEnd;

    } NTFS_VOLUME_DATA_BUFFER, *PNTFS_VOLUME_DATA_BUFFER;


  • [Python 3]: struct - Interpret bytes as packed binary data module, is used for conversions between binary and "normal" data. It contains all the format characters meanings (q, Q, L, ...), and much more. You could also take a look at [SO]: Python struct.pack() behavior for more (practical) details



After going over the above materials, things should become clearer.



A couple of notes:




  • If one doesn't know what a function does (returns), they should probably don't use it (without reading the manual, of course). Although nowadays, both Win (which always had a lot of restrictions for the regular user) and Ux "protect users from themselves" (e.g.: root login no longer allowed, write protect %SystemDrive%)

  • The attempts (trial and error) show some lack of experience (probably every does it at some point, the key is not to rely solely on it)

  • "Would I have to remember these formats every-time I want to get something out from DeviceIoControl"?


    • Again, if not knowing that a function does, what's the reason for calling it? If you meant learning NTFS_VOLUME_DATA_BUFFER by heart, it's definitely not the case. You should know its structure only when using it (and as you've noticed there are some places that you can get it from - including this very post :) )



  • "My understanding was we can unpack if buffer was larger than the output so why doesn't the unpack work?"


    • Your understanding is correct. But win32file.DeviceIoControl seems to sometimes (probably when reaching 1stNULL after 96 bytes) truncate the output buffer when passing a value greater than the expected one (via length argument). When passing a smaller one, it will fail (as expected)




I've also prepared a dummy Python example.



code.py:



#!/usr/bin/env python3

import sys
import struct
import win32file
import win32api
import win32con
import winioctlcon


VOLUME_LETTER = "E"

FILE_READ_ATTRIBUTES = 0x0080
FILE_EXECUTE = 0x0020

vol_data_buf_fmt = "qqqqqLLLLqqqqq" # This is the format that matches NTFS_VOLUME_DATA_BUFFER definition (96 bytes). Note: Instead of each 'q' you could also use 'Ll' as 'LARGE_INTEGER' is an union

BINARY_FORMAT_LIST = [
vol_data_buf_fmt,
"QQQQQQQQQQQQQQQQQQQ",
]


def print_formats(): # Dummy func
print("Formats and lengths:")
for format in BINARY_FORMAT_LIST:
print(" {:s}: {:d}".format(format, struct.calcsize(format)))


def main():
#print_formats()
vol_unc_name = "\\.\{:s}:".format(VOLUME_LETTER)
print("volume: ", vol_unc_name)
access_flags = FILE_READ_ATTRIBUTES | FILE_EXECUTE # Apparently, doesn't work without FILE_EXECUTE
share_flags = win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE # Doesn't work withou FILE_SHARE_WRITE
creation_flags = win32con.OPEN_EXISTING
attributes_flags = win32con.FILE_ATTRIBUTE_NORMAL
vol_handle = win32file.CreateFile(vol_unc_name, access_flags, share_flags, None, creation_flags, attributes_flags, None)

buf_len = struct.calcsize(vol_data_buf_fmt)
for i in [buf_len]:
print(" Passing a buffer size of: {:d}".format(i))
buf = win32file.DeviceIoControl(vol_handle, winioctlcon.FSCTL_GET_NTFS_VOLUME_DATA, None, i)
print(" DeviceIocontrol returned a {:d} bytes long {:}".format(len(buf), type(buf)))
out = struct.unpack_from(vol_data_buf_fmt, buf)
print("n NumberSectors: {:}n TotalClusters: {:}n BytesPerCluster: {:}".format(out[1], out[2], out[6]))
win32api.CloseHandle(vol_handle)


if __name__ == "__main__":
print("Python {:s} on {:s}n".format(sys.version, sys.platform))
main()


Output:




(py35x64_test) e:WorkDevStackOverflowq053318932>"e:WorkDevVEnvspy35x64_testScriptspython.exe" ./code.py
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

volume: \.E:
Passing a buffer size of: 96
DeviceIocontrol returned a 96 bytes long <class 'bytes'>

NumberSectors: 494374911
TotalClusters: 61796863
BytesPerCluster: 4096



Needless to say that multiplying TotalClusters by BytesPerCluster, I get the correct bytes number (as reported by Win) for my E: drive.






share|improve this answer


























  • Thanks @CristiFati. Actually, with a bit of hit and try and research, I was able to find the resources mentioned in your comment and was thinking of answering my own question if nobody answered it but you have

    – ThatBird
    Nov 19 '18 at 10:32











  • "protect users from themselves" as you said so I knew the risks I was getting into while poking around with these commands

    – ThatBird
    Nov 19 '18 at 10:32











  • "Every attempt simply expresses lack of experience" but attempt adds up to the experience, which in turn helped me to discover these docs

    – ThatBird
    Nov 19 '18 at 10:34











  • I can query through the journal now but I need to build path to parent file, this is where I am stuck currently. What I have found out is there are to ways to get this to work, one is going through the MFT(I don't want that) and the other is opening file with File Reference Number. Can you help me with the second one?

    – ThatBird
    Nov 19 '18 at 10:37











  • I'm afraid that my knowledge on "File Reference Number". Is somewhere close to 0. So without some code samples I can't do anything. But, wouldn't that be a different question (cause the code from this one is related to volume properties rather than volume changes)?

    – CristiFati
    Nov 19 '18 at 13:33


















2














Let's split the problem in smaller pieces and take each one at a time.





  • win32file module is part of [GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions which is a Python wrapper over WinAPIs




    • Unfortunately, it doesn't have an official doc page (or I'm not aware of one), so below is the best that I could find (I've been using this for years). An alternative method that never fails (but it's less appealing) is looking directly at the code


    • [ActiveState]: win32file.DeviceIoControl is a wrapper over [MS.Docs]: DeviceIoControl function





  • DeviceIoControl acts differently, depending on the dwIoControlCode (2nd argument). For FSCTL_GET_NTFS_VOLUME_DATA, it fills a buffer with volume specific data. From [MSDN]: FSCTL_GET_NTFS_VOLUME_DATA control code:




    lpOutBuffer
    A pointer to the output buffer, an NTFS_VOLUME_DATA_BUFFER (@CristiFati: !!! Broken URL !!!) structure. The file record associated with the file identifier specified in the input buffer is returned in this buffer. Refer to the Remarks section of the documentation for the NTFS_VOLUME_DATA_BUFFER structure for specific information on how to determine the correct size of this buffer.




    Here's an alternative to the above broken URL: [MSDN]: NTFS_VOLUME_DATA_BUFFER structure. As I'm not sure for how long will it be valid, I'm pasting the structure definition below (from Windows Kits 8.1: winioctl.h (line #4987)):



    typedef struct {

    LARGE_INTEGER VolumeSerialNumber;
    LARGE_INTEGER NumberSectors;
    LARGE_INTEGER TotalClusters;
    LARGE_INTEGER FreeClusters;
    LARGE_INTEGER TotalReserved;
    DWORD BytesPerSector;
    DWORD BytesPerCluster;
    DWORD BytesPerFileRecordSegment;
    DWORD ClustersPerFileRecordSegment;
    LARGE_INTEGER MftValidDataLength;
    LARGE_INTEGER MftStartLcn;
    LARGE_INTEGER Mft2StartLcn;
    LARGE_INTEGER MftZoneStart;
    LARGE_INTEGER MftZoneEnd;

    } NTFS_VOLUME_DATA_BUFFER, *PNTFS_VOLUME_DATA_BUFFER;


  • [Python 3]: struct - Interpret bytes as packed binary data module, is used for conversions between binary and "normal" data. It contains all the format characters meanings (q, Q, L, ...), and much more. You could also take a look at [SO]: Python struct.pack() behavior for more (practical) details



After going over the above materials, things should become clearer.



A couple of notes:




  • If one doesn't know what a function does (returns), they should probably don't use it (without reading the manual, of course). Although nowadays, both Win (which always had a lot of restrictions for the regular user) and Ux "protect users from themselves" (e.g.: root login no longer allowed, write protect %SystemDrive%)

  • The attempts (trial and error) show some lack of experience (probably every does it at some point, the key is not to rely solely on it)

  • "Would I have to remember these formats every-time I want to get something out from DeviceIoControl"?


    • Again, if not knowing that a function does, what's the reason for calling it? If you meant learning NTFS_VOLUME_DATA_BUFFER by heart, it's definitely not the case. You should know its structure only when using it (and as you've noticed there are some places that you can get it from - including this very post :) )



  • "My understanding was we can unpack if buffer was larger than the output so why doesn't the unpack work?"


    • Your understanding is correct. But win32file.DeviceIoControl seems to sometimes (probably when reaching 1stNULL after 96 bytes) truncate the output buffer when passing a value greater than the expected one (via length argument). When passing a smaller one, it will fail (as expected)




I've also prepared a dummy Python example.



code.py:



#!/usr/bin/env python3

import sys
import struct
import win32file
import win32api
import win32con
import winioctlcon


VOLUME_LETTER = "E"

FILE_READ_ATTRIBUTES = 0x0080
FILE_EXECUTE = 0x0020

vol_data_buf_fmt = "qqqqqLLLLqqqqq" # This is the format that matches NTFS_VOLUME_DATA_BUFFER definition (96 bytes). Note: Instead of each 'q' you could also use 'Ll' as 'LARGE_INTEGER' is an union

BINARY_FORMAT_LIST = [
vol_data_buf_fmt,
"QQQQQQQQQQQQQQQQQQQ",
]


def print_formats(): # Dummy func
print("Formats and lengths:")
for format in BINARY_FORMAT_LIST:
print(" {:s}: {:d}".format(format, struct.calcsize(format)))


def main():
#print_formats()
vol_unc_name = "\\.\{:s}:".format(VOLUME_LETTER)
print("volume: ", vol_unc_name)
access_flags = FILE_READ_ATTRIBUTES | FILE_EXECUTE # Apparently, doesn't work without FILE_EXECUTE
share_flags = win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE # Doesn't work withou FILE_SHARE_WRITE
creation_flags = win32con.OPEN_EXISTING
attributes_flags = win32con.FILE_ATTRIBUTE_NORMAL
vol_handle = win32file.CreateFile(vol_unc_name, access_flags, share_flags, None, creation_flags, attributes_flags, None)

buf_len = struct.calcsize(vol_data_buf_fmt)
for i in [buf_len]:
print(" Passing a buffer size of: {:d}".format(i))
buf = win32file.DeviceIoControl(vol_handle, winioctlcon.FSCTL_GET_NTFS_VOLUME_DATA, None, i)
print(" DeviceIocontrol returned a {:d} bytes long {:}".format(len(buf), type(buf)))
out = struct.unpack_from(vol_data_buf_fmt, buf)
print("n NumberSectors: {:}n TotalClusters: {:}n BytesPerCluster: {:}".format(out[1], out[2], out[6]))
win32api.CloseHandle(vol_handle)


if __name__ == "__main__":
print("Python {:s} on {:s}n".format(sys.version, sys.platform))
main()


Output:




(py35x64_test) e:WorkDevStackOverflowq053318932>"e:WorkDevVEnvspy35x64_testScriptspython.exe" ./code.py
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

volume: \.E:
Passing a buffer size of: 96
DeviceIocontrol returned a 96 bytes long <class 'bytes'>

NumberSectors: 494374911
TotalClusters: 61796863
BytesPerCluster: 4096



Needless to say that multiplying TotalClusters by BytesPerCluster, I get the correct bytes number (as reported by Win) for my E: drive.






share|improve this answer


























  • Thanks @CristiFati. Actually, with a bit of hit and try and research, I was able to find the resources mentioned in your comment and was thinking of answering my own question if nobody answered it but you have

    – ThatBird
    Nov 19 '18 at 10:32











  • "protect users from themselves" as you said so I knew the risks I was getting into while poking around with these commands

    – ThatBird
    Nov 19 '18 at 10:32











  • "Every attempt simply expresses lack of experience" but attempt adds up to the experience, which in turn helped me to discover these docs

    – ThatBird
    Nov 19 '18 at 10:34











  • I can query through the journal now but I need to build path to parent file, this is where I am stuck currently. What I have found out is there are to ways to get this to work, one is going through the MFT(I don't want that) and the other is opening file with File Reference Number. Can you help me with the second one?

    – ThatBird
    Nov 19 '18 at 10:37











  • I'm afraid that my knowledge on "File Reference Number". Is somewhere close to 0. So without some code samples I can't do anything. But, wouldn't that be a different question (cause the code from this one is related to volume properties rather than volume changes)?

    – CristiFati
    Nov 19 '18 at 13:33
















2












2








2







Let's split the problem in smaller pieces and take each one at a time.





  • win32file module is part of [GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions which is a Python wrapper over WinAPIs




    • Unfortunately, it doesn't have an official doc page (or I'm not aware of one), so below is the best that I could find (I've been using this for years). An alternative method that never fails (but it's less appealing) is looking directly at the code


    • [ActiveState]: win32file.DeviceIoControl is a wrapper over [MS.Docs]: DeviceIoControl function





  • DeviceIoControl acts differently, depending on the dwIoControlCode (2nd argument). For FSCTL_GET_NTFS_VOLUME_DATA, it fills a buffer with volume specific data. From [MSDN]: FSCTL_GET_NTFS_VOLUME_DATA control code:




    lpOutBuffer
    A pointer to the output buffer, an NTFS_VOLUME_DATA_BUFFER (@CristiFati: !!! Broken URL !!!) structure. The file record associated with the file identifier specified in the input buffer is returned in this buffer. Refer to the Remarks section of the documentation for the NTFS_VOLUME_DATA_BUFFER structure for specific information on how to determine the correct size of this buffer.




    Here's an alternative to the above broken URL: [MSDN]: NTFS_VOLUME_DATA_BUFFER structure. As I'm not sure for how long will it be valid, I'm pasting the structure definition below (from Windows Kits 8.1: winioctl.h (line #4987)):



    typedef struct {

    LARGE_INTEGER VolumeSerialNumber;
    LARGE_INTEGER NumberSectors;
    LARGE_INTEGER TotalClusters;
    LARGE_INTEGER FreeClusters;
    LARGE_INTEGER TotalReserved;
    DWORD BytesPerSector;
    DWORD BytesPerCluster;
    DWORD BytesPerFileRecordSegment;
    DWORD ClustersPerFileRecordSegment;
    LARGE_INTEGER MftValidDataLength;
    LARGE_INTEGER MftStartLcn;
    LARGE_INTEGER Mft2StartLcn;
    LARGE_INTEGER MftZoneStart;
    LARGE_INTEGER MftZoneEnd;

    } NTFS_VOLUME_DATA_BUFFER, *PNTFS_VOLUME_DATA_BUFFER;


  • [Python 3]: struct - Interpret bytes as packed binary data module, is used for conversions between binary and "normal" data. It contains all the format characters meanings (q, Q, L, ...), and much more. You could also take a look at [SO]: Python struct.pack() behavior for more (practical) details



After going over the above materials, things should become clearer.



A couple of notes:




  • If one doesn't know what a function does (returns), they should probably don't use it (without reading the manual, of course). Although nowadays, both Win (which always had a lot of restrictions for the regular user) and Ux "protect users from themselves" (e.g.: root login no longer allowed, write protect %SystemDrive%)

  • The attempts (trial and error) show some lack of experience (probably every does it at some point, the key is not to rely solely on it)

  • "Would I have to remember these formats every-time I want to get something out from DeviceIoControl"?


    • Again, if not knowing that a function does, what's the reason for calling it? If you meant learning NTFS_VOLUME_DATA_BUFFER by heart, it's definitely not the case. You should know its structure only when using it (and as you've noticed there are some places that you can get it from - including this very post :) )



  • "My understanding was we can unpack if buffer was larger than the output so why doesn't the unpack work?"


    • Your understanding is correct. But win32file.DeviceIoControl seems to sometimes (probably when reaching 1stNULL after 96 bytes) truncate the output buffer when passing a value greater than the expected one (via length argument). When passing a smaller one, it will fail (as expected)




I've also prepared a dummy Python example.



code.py:



#!/usr/bin/env python3

import sys
import struct
import win32file
import win32api
import win32con
import winioctlcon


VOLUME_LETTER = "E"

FILE_READ_ATTRIBUTES = 0x0080
FILE_EXECUTE = 0x0020

vol_data_buf_fmt = "qqqqqLLLLqqqqq" # This is the format that matches NTFS_VOLUME_DATA_BUFFER definition (96 bytes). Note: Instead of each 'q' you could also use 'Ll' as 'LARGE_INTEGER' is an union

BINARY_FORMAT_LIST = [
vol_data_buf_fmt,
"QQQQQQQQQQQQQQQQQQQ",
]


def print_formats(): # Dummy func
print("Formats and lengths:")
for format in BINARY_FORMAT_LIST:
print(" {:s}: {:d}".format(format, struct.calcsize(format)))


def main():
#print_formats()
vol_unc_name = "\\.\{:s}:".format(VOLUME_LETTER)
print("volume: ", vol_unc_name)
access_flags = FILE_READ_ATTRIBUTES | FILE_EXECUTE # Apparently, doesn't work without FILE_EXECUTE
share_flags = win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE # Doesn't work withou FILE_SHARE_WRITE
creation_flags = win32con.OPEN_EXISTING
attributes_flags = win32con.FILE_ATTRIBUTE_NORMAL
vol_handle = win32file.CreateFile(vol_unc_name, access_flags, share_flags, None, creation_flags, attributes_flags, None)

buf_len = struct.calcsize(vol_data_buf_fmt)
for i in [buf_len]:
print(" Passing a buffer size of: {:d}".format(i))
buf = win32file.DeviceIoControl(vol_handle, winioctlcon.FSCTL_GET_NTFS_VOLUME_DATA, None, i)
print(" DeviceIocontrol returned a {:d} bytes long {:}".format(len(buf), type(buf)))
out = struct.unpack_from(vol_data_buf_fmt, buf)
print("n NumberSectors: {:}n TotalClusters: {:}n BytesPerCluster: {:}".format(out[1], out[2], out[6]))
win32api.CloseHandle(vol_handle)


if __name__ == "__main__":
print("Python {:s} on {:s}n".format(sys.version, sys.platform))
main()


Output:




(py35x64_test) e:WorkDevStackOverflowq053318932>"e:WorkDevVEnvspy35x64_testScriptspython.exe" ./code.py
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

volume: \.E:
Passing a buffer size of: 96
DeviceIocontrol returned a 96 bytes long <class 'bytes'>

NumberSectors: 494374911
TotalClusters: 61796863
BytesPerCluster: 4096



Needless to say that multiplying TotalClusters by BytesPerCluster, I get the correct bytes number (as reported by Win) for my E: drive.






share|improve this answer















Let's split the problem in smaller pieces and take each one at a time.





  • win32file module is part of [GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions which is a Python wrapper over WinAPIs




    • Unfortunately, it doesn't have an official doc page (or I'm not aware of one), so below is the best that I could find (I've been using this for years). An alternative method that never fails (but it's less appealing) is looking directly at the code


    • [ActiveState]: win32file.DeviceIoControl is a wrapper over [MS.Docs]: DeviceIoControl function





  • DeviceIoControl acts differently, depending on the dwIoControlCode (2nd argument). For FSCTL_GET_NTFS_VOLUME_DATA, it fills a buffer with volume specific data. From [MSDN]: FSCTL_GET_NTFS_VOLUME_DATA control code:




    lpOutBuffer
    A pointer to the output buffer, an NTFS_VOLUME_DATA_BUFFER (@CristiFati: !!! Broken URL !!!) structure. The file record associated with the file identifier specified in the input buffer is returned in this buffer. Refer to the Remarks section of the documentation for the NTFS_VOLUME_DATA_BUFFER structure for specific information on how to determine the correct size of this buffer.




    Here's an alternative to the above broken URL: [MSDN]: NTFS_VOLUME_DATA_BUFFER structure. As I'm not sure for how long will it be valid, I'm pasting the structure definition below (from Windows Kits 8.1: winioctl.h (line #4987)):



    typedef struct {

    LARGE_INTEGER VolumeSerialNumber;
    LARGE_INTEGER NumberSectors;
    LARGE_INTEGER TotalClusters;
    LARGE_INTEGER FreeClusters;
    LARGE_INTEGER TotalReserved;
    DWORD BytesPerSector;
    DWORD BytesPerCluster;
    DWORD BytesPerFileRecordSegment;
    DWORD ClustersPerFileRecordSegment;
    LARGE_INTEGER MftValidDataLength;
    LARGE_INTEGER MftStartLcn;
    LARGE_INTEGER Mft2StartLcn;
    LARGE_INTEGER MftZoneStart;
    LARGE_INTEGER MftZoneEnd;

    } NTFS_VOLUME_DATA_BUFFER, *PNTFS_VOLUME_DATA_BUFFER;


  • [Python 3]: struct - Interpret bytes as packed binary data module, is used for conversions between binary and "normal" data. It contains all the format characters meanings (q, Q, L, ...), and much more. You could also take a look at [SO]: Python struct.pack() behavior for more (practical) details



After going over the above materials, things should become clearer.



A couple of notes:




  • If one doesn't know what a function does (returns), they should probably don't use it (without reading the manual, of course). Although nowadays, both Win (which always had a lot of restrictions for the regular user) and Ux "protect users from themselves" (e.g.: root login no longer allowed, write protect %SystemDrive%)

  • The attempts (trial and error) show some lack of experience (probably every does it at some point, the key is not to rely solely on it)

  • "Would I have to remember these formats every-time I want to get something out from DeviceIoControl"?


    • Again, if not knowing that a function does, what's the reason for calling it? If you meant learning NTFS_VOLUME_DATA_BUFFER by heart, it's definitely not the case. You should know its structure only when using it (and as you've noticed there are some places that you can get it from - including this very post :) )



  • "My understanding was we can unpack if buffer was larger than the output so why doesn't the unpack work?"


    • Your understanding is correct. But win32file.DeviceIoControl seems to sometimes (probably when reaching 1stNULL after 96 bytes) truncate the output buffer when passing a value greater than the expected one (via length argument). When passing a smaller one, it will fail (as expected)




I've also prepared a dummy Python example.



code.py:



#!/usr/bin/env python3

import sys
import struct
import win32file
import win32api
import win32con
import winioctlcon


VOLUME_LETTER = "E"

FILE_READ_ATTRIBUTES = 0x0080
FILE_EXECUTE = 0x0020

vol_data_buf_fmt = "qqqqqLLLLqqqqq" # This is the format that matches NTFS_VOLUME_DATA_BUFFER definition (96 bytes). Note: Instead of each 'q' you could also use 'Ll' as 'LARGE_INTEGER' is an union

BINARY_FORMAT_LIST = [
vol_data_buf_fmt,
"QQQQQQQQQQQQQQQQQQQ",
]


def print_formats(): # Dummy func
print("Formats and lengths:")
for format in BINARY_FORMAT_LIST:
print(" {:s}: {:d}".format(format, struct.calcsize(format)))


def main():
#print_formats()
vol_unc_name = "\\.\{:s}:".format(VOLUME_LETTER)
print("volume: ", vol_unc_name)
access_flags = FILE_READ_ATTRIBUTES | FILE_EXECUTE # Apparently, doesn't work without FILE_EXECUTE
share_flags = win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE # Doesn't work withou FILE_SHARE_WRITE
creation_flags = win32con.OPEN_EXISTING
attributes_flags = win32con.FILE_ATTRIBUTE_NORMAL
vol_handle = win32file.CreateFile(vol_unc_name, access_flags, share_flags, None, creation_flags, attributes_flags, None)

buf_len = struct.calcsize(vol_data_buf_fmt)
for i in [buf_len]:
print(" Passing a buffer size of: {:d}".format(i))
buf = win32file.DeviceIoControl(vol_handle, winioctlcon.FSCTL_GET_NTFS_VOLUME_DATA, None, i)
print(" DeviceIocontrol returned a {:d} bytes long {:}".format(len(buf), type(buf)))
out = struct.unpack_from(vol_data_buf_fmt, buf)
print("n NumberSectors: {:}n TotalClusters: {:}n BytesPerCluster: {:}".format(out[1], out[2], out[6]))
win32api.CloseHandle(vol_handle)


if __name__ == "__main__":
print("Python {:s} on {:s}n".format(sys.version, sys.platform))
main()


Output:




(py35x64_test) e:WorkDevStackOverflowq053318932>"e:WorkDevVEnvspy35x64_testScriptspython.exe" ./code.py
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

volume: \.E:
Passing a buffer size of: 96
DeviceIocontrol returned a 96 bytes long <class 'bytes'>

NumberSectors: 494374911
TotalClusters: 61796863
BytesPerCluster: 4096



Needless to say that multiplying TotalClusters by BytesPerCluster, I get the correct bytes number (as reported by Win) for my E: drive.







share|improve this answer














share|improve this answer



share|improve this answer








edited Dec 14 '18 at 2:33

























answered Nov 17 '18 at 1:30









CristiFatiCristiFati

13.9k72437




13.9k72437













  • Thanks @CristiFati. Actually, with a bit of hit and try and research, I was able to find the resources mentioned in your comment and was thinking of answering my own question if nobody answered it but you have

    – ThatBird
    Nov 19 '18 at 10:32











  • "protect users from themselves" as you said so I knew the risks I was getting into while poking around with these commands

    – ThatBird
    Nov 19 '18 at 10:32











  • "Every attempt simply expresses lack of experience" but attempt adds up to the experience, which in turn helped me to discover these docs

    – ThatBird
    Nov 19 '18 at 10:34











  • I can query through the journal now but I need to build path to parent file, this is where I am stuck currently. What I have found out is there are to ways to get this to work, one is going through the MFT(I don't want that) and the other is opening file with File Reference Number. Can you help me with the second one?

    – ThatBird
    Nov 19 '18 at 10:37











  • I'm afraid that my knowledge on "File Reference Number". Is somewhere close to 0. So without some code samples I can't do anything. But, wouldn't that be a different question (cause the code from this one is related to volume properties rather than volume changes)?

    – CristiFati
    Nov 19 '18 at 13:33





















  • Thanks @CristiFati. Actually, with a bit of hit and try and research, I was able to find the resources mentioned in your comment and was thinking of answering my own question if nobody answered it but you have

    – ThatBird
    Nov 19 '18 at 10:32











  • "protect users from themselves" as you said so I knew the risks I was getting into while poking around with these commands

    – ThatBird
    Nov 19 '18 at 10:32











  • "Every attempt simply expresses lack of experience" but attempt adds up to the experience, which in turn helped me to discover these docs

    – ThatBird
    Nov 19 '18 at 10:34











  • I can query through the journal now but I need to build path to parent file, this is where I am stuck currently. What I have found out is there are to ways to get this to work, one is going through the MFT(I don't want that) and the other is opening file with File Reference Number. Can you help me with the second one?

    – ThatBird
    Nov 19 '18 at 10:37











  • I'm afraid that my knowledge on "File Reference Number". Is somewhere close to 0. So without some code samples I can't do anything. But, wouldn't that be a different question (cause the code from this one is related to volume properties rather than volume changes)?

    – CristiFati
    Nov 19 '18 at 13:33



















Thanks @CristiFati. Actually, with a bit of hit and try and research, I was able to find the resources mentioned in your comment and was thinking of answering my own question if nobody answered it but you have

– ThatBird
Nov 19 '18 at 10:32





Thanks @CristiFati. Actually, with a bit of hit and try and research, I was able to find the resources mentioned in your comment and was thinking of answering my own question if nobody answered it but you have

– ThatBird
Nov 19 '18 at 10:32













"protect users from themselves" as you said so I knew the risks I was getting into while poking around with these commands

– ThatBird
Nov 19 '18 at 10:32





"protect users from themselves" as you said so I knew the risks I was getting into while poking around with these commands

– ThatBird
Nov 19 '18 at 10:32













"Every attempt simply expresses lack of experience" but attempt adds up to the experience, which in turn helped me to discover these docs

– ThatBird
Nov 19 '18 at 10:34





"Every attempt simply expresses lack of experience" but attempt adds up to the experience, which in turn helped me to discover these docs

– ThatBird
Nov 19 '18 at 10:34













I can query through the journal now but I need to build path to parent file, this is where I am stuck currently. What I have found out is there are to ways to get this to work, one is going through the MFT(I don't want that) and the other is opening file with File Reference Number. Can you help me with the second one?

– ThatBird
Nov 19 '18 at 10:37





I can query through the journal now but I need to build path to parent file, this is where I am stuck currently. What I have found out is there are to ways to get this to work, one is going through the MFT(I don't want that) and the other is opening file with File Reference Number. Can you help me with the second one?

– ThatBird
Nov 19 '18 at 10:37













I'm afraid that my knowledge on "File Reference Number". Is somewhere close to 0. So without some code samples I can't do anything. But, wouldn't that be a different question (cause the code from this one is related to volume properties rather than volume changes)?

– CristiFati
Nov 19 '18 at 13:33







I'm afraid that my knowledge on "File Reference Number". Is somewhere close to 0. So without some code samples I can't do anything. But, wouldn't that be a different question (cause the code from this one is related to volume properties rather than volume changes)?

– CristiFati
Nov 19 '18 at 13:33






















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53318932%2fstruct-unpack-on-win32file-deviceiocontrol%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Bressuire

Vorschmack

Quarantine