update aTable set a,b,c = func(x,y,z,…)












0















I need a quick advice how-to. I mention that the following scenario is based on the use of c_api available already to my monetdblite compilation on 64bit, intention is to use it with some adhoc C written functions.



Short: how can I achieve or simulate the following scenario:
update aTable set a,b,c = func(x,y,z,…)



Long. Many algorithms are returning more than one variable as, for instance, multiple regression.



bool m_regression(IN const double **data, IN const int cols, IN const int rows, OUT double *fit_values, OUT double *residuals, OUT double *std_residuals, OUT double &p_value);


In order to minimize the transfer of data between monetdb and heavy computational function, all those results are generated in one step. Question is how can I transfer them back at once, minimizing computational time and memory traffic between monetdb and external C/C++(/R/Python) function?





My first thought to solve this is something like this:



1. update aTable set dummy = func_compute(x,y,z,…)



where dummy is a temporary __int64 field and func_compute will compute all the necessary outputs and store the result into a dummy pointer. To make sure is no issue with constant estimation, first returned value in the array will be the real dummy pointer, the rest just an incremented value of dummy + i;



2. update aTable set a = func_ret(dummy, 1), b= func_ret (dummy, 2), c= func_ret (dummy, 3) [, dummy=func_free(dummy)];



Assuming the func_ret will get the dummy in the same order that it was returned on first call, I would just copy the prepared result into provided storage; In case the order is not preserved, I will need an extra step to get the minimum (real dummy pointer), then to use the offset of current value to lookup in my array.



__int64 real_dummy = __inputs[0][0];

double *my_pointer_data = (double *) (real_dummy + __inputs[1][0] * sizeof(double)* row_count);

memcpy(__outputs[0], my_pointer_data, sizeof(double)* row_count);


// or ============================



__int64 real_dummy = minimum(__inputs[0]);

double *my_pointer_data = (double *) (real_dummy + __inputs[0][1] * sizeof(double)* row_count);

for (int i=0;i<row_count;i++)
__outputs[0][i] = my_pointer_data[__inputs[0][i] - real_dummy];


It is less relevant how am I going to free the temporary memory, can be in the last statement in update or in a new fake update statement using func_free.
Problem is that it doesn’t look to me that, even if I save some computational (big) time, the passing of the dummy is still done 3 times (any chance that memory is actually not copied?).



Is it any other better way of achieving this?










share|improve this question



























    0















    I need a quick advice how-to. I mention that the following scenario is based on the use of c_api available already to my monetdblite compilation on 64bit, intention is to use it with some adhoc C written functions.



    Short: how can I achieve or simulate the following scenario:
    update aTable set a,b,c = func(x,y,z,…)



    Long. Many algorithms are returning more than one variable as, for instance, multiple regression.



    bool m_regression(IN const double **data, IN const int cols, IN const int rows, OUT double *fit_values, OUT double *residuals, OUT double *std_residuals, OUT double &p_value);


    In order to minimize the transfer of data between monetdb and heavy computational function, all those results are generated in one step. Question is how can I transfer them back at once, minimizing computational time and memory traffic between monetdb and external C/C++(/R/Python) function?





    My first thought to solve this is something like this:



    1. update aTable set dummy = func_compute(x,y,z,…)



    where dummy is a temporary __int64 field and func_compute will compute all the necessary outputs and store the result into a dummy pointer. To make sure is no issue with constant estimation, first returned value in the array will be the real dummy pointer, the rest just an incremented value of dummy + i;



    2. update aTable set a = func_ret(dummy, 1), b= func_ret (dummy, 2), c= func_ret (dummy, 3) [, dummy=func_free(dummy)];



    Assuming the func_ret will get the dummy in the same order that it was returned on first call, I would just copy the prepared result into provided storage; In case the order is not preserved, I will need an extra step to get the minimum (real dummy pointer), then to use the offset of current value to lookup in my array.



    __int64 real_dummy = __inputs[0][0];

    double *my_pointer_data = (double *) (real_dummy + __inputs[1][0] * sizeof(double)* row_count);

    memcpy(__outputs[0], my_pointer_data, sizeof(double)* row_count);


    // or ============================



    __int64 real_dummy = minimum(__inputs[0]);

    double *my_pointer_data = (double *) (real_dummy + __inputs[0][1] * sizeof(double)* row_count);

    for (int i=0;i<row_count;i++)
    __outputs[0][i] = my_pointer_data[__inputs[0][i] - real_dummy];


    It is less relevant how am I going to free the temporary memory, can be in the last statement in update or in a new fake update statement using func_free.
    Problem is that it doesn’t look to me that, even if I save some computational (big) time, the passing of the dummy is still done 3 times (any chance that memory is actually not copied?).



    Is it any other better way of achieving this?










    share|improve this question

























      0












      0








      0








      I need a quick advice how-to. I mention that the following scenario is based on the use of c_api available already to my monetdblite compilation on 64bit, intention is to use it with some adhoc C written functions.



      Short: how can I achieve or simulate the following scenario:
      update aTable set a,b,c = func(x,y,z,…)



      Long. Many algorithms are returning more than one variable as, for instance, multiple regression.



      bool m_regression(IN const double **data, IN const int cols, IN const int rows, OUT double *fit_values, OUT double *residuals, OUT double *std_residuals, OUT double &p_value);


      In order to minimize the transfer of data between monetdb and heavy computational function, all those results are generated in one step. Question is how can I transfer them back at once, minimizing computational time and memory traffic between monetdb and external C/C++(/R/Python) function?





      My first thought to solve this is something like this:



      1. update aTable set dummy = func_compute(x,y,z,…)



      where dummy is a temporary __int64 field and func_compute will compute all the necessary outputs and store the result into a dummy pointer. To make sure is no issue with constant estimation, first returned value in the array will be the real dummy pointer, the rest just an incremented value of dummy + i;



      2. update aTable set a = func_ret(dummy, 1), b= func_ret (dummy, 2), c= func_ret (dummy, 3) [, dummy=func_free(dummy)];



      Assuming the func_ret will get the dummy in the same order that it was returned on first call, I would just copy the prepared result into provided storage; In case the order is not preserved, I will need an extra step to get the minimum (real dummy pointer), then to use the offset of current value to lookup in my array.



      __int64 real_dummy = __inputs[0][0];

      double *my_pointer_data = (double *) (real_dummy + __inputs[1][0] * sizeof(double)* row_count);

      memcpy(__outputs[0], my_pointer_data, sizeof(double)* row_count);


      // or ============================



      __int64 real_dummy = minimum(__inputs[0]);

      double *my_pointer_data = (double *) (real_dummy + __inputs[0][1] * sizeof(double)* row_count);

      for (int i=0;i<row_count;i++)
      __outputs[0][i] = my_pointer_data[__inputs[0][i] - real_dummy];


      It is less relevant how am I going to free the temporary memory, can be in the last statement in update or in a new fake update statement using func_free.
      Problem is that it doesn’t look to me that, even if I save some computational (big) time, the passing of the dummy is still done 3 times (any chance that memory is actually not copied?).



      Is it any other better way of achieving this?










      share|improve this question














      I need a quick advice how-to. I mention that the following scenario is based on the use of c_api available already to my monetdblite compilation on 64bit, intention is to use it with some adhoc C written functions.



      Short: how can I achieve or simulate the following scenario:
      update aTable set a,b,c = func(x,y,z,…)



      Long. Many algorithms are returning more than one variable as, for instance, multiple regression.



      bool m_regression(IN const double **data, IN const int cols, IN const int rows, OUT double *fit_values, OUT double *residuals, OUT double *std_residuals, OUT double &p_value);


      In order to minimize the transfer of data between monetdb and heavy computational function, all those results are generated in one step. Question is how can I transfer them back at once, minimizing computational time and memory traffic between monetdb and external C/C++(/R/Python) function?





      My first thought to solve this is something like this:



      1. update aTable set dummy = func_compute(x,y,z,…)



      where dummy is a temporary __int64 field and func_compute will compute all the necessary outputs and store the result into a dummy pointer. To make sure is no issue with constant estimation, first returned value in the array will be the real dummy pointer, the rest just an incremented value of dummy + i;



      2. update aTable set a = func_ret(dummy, 1), b= func_ret (dummy, 2), c= func_ret (dummy, 3) [, dummy=func_free(dummy)];



      Assuming the func_ret will get the dummy in the same order that it was returned on first call, I would just copy the prepared result into provided storage; In case the order is not preserved, I will need an extra step to get the minimum (real dummy pointer), then to use the offset of current value to lookup in my array.



      __int64 real_dummy = __inputs[0][0];

      double *my_pointer_data = (double *) (real_dummy + __inputs[1][0] * sizeof(double)* row_count);

      memcpy(__outputs[0], my_pointer_data, sizeof(double)* row_count);


      // or ============================



      __int64 real_dummy = minimum(__inputs[0]);

      double *my_pointer_data = (double *) (real_dummy + __inputs[0][1] * sizeof(double)* row_count);

      for (int i=0;i<row_count;i++)
      __outputs[0][i] = my_pointer_data[__inputs[0][i] - real_dummy];


      It is less relevant how am I going to free the temporary memory, can be in the last statement in update or in a new fake update statement using func_free.
      Problem is that it doesn’t look to me that, even if I save some computational (big) time, the passing of the dummy is still done 3 times (any chance that memory is actually not copied?).



      Is it any other better way of achieving this?







      c-api monetdblite






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 15 '18 at 11:45









      DanZDanZ

      104




      104
























          1 Answer
          1






          active

          oldest

          votes


















          0














          I am not aware of a good way of doing this, sorry. You could retrieve the table, add your columns as BATs in whichever way you like and write it back.






          share|improve this answer























            Your Answer






            StackExchange.ifUsing("editor", function () {
            StackExchange.using("externalEditor", function () {
            StackExchange.using("snippets", function () {
            StackExchange.snippets.init();
            });
            });
            }, "code-snippets");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "1"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53318777%2fupdate-atable-set-a-b-c-funcx-y-z%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            0














            I am not aware of a good way of doing this, sorry. You could retrieve the table, add your columns as BATs in whichever way you like and write it back.






            share|improve this answer




























              0














              I am not aware of a good way of doing this, sorry. You could retrieve the table, add your columns as BATs in whichever way you like and write it back.






              share|improve this answer


























                0












                0








                0







                I am not aware of a good way of doing this, sorry. You could retrieve the table, add your columns as BATs in whichever way you like and write it back.






                share|improve this answer













                I am not aware of a good way of doing this, sorry. You could retrieve the table, add your columns as BATs in whichever way you like and write it back.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Jan 31 at 7:57









                Hannes MühleisenHannes Mühleisen

                2,267513




                2,267513
































                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Stack Overflow!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53318777%2fupdate-atable-set-a-b-c-funcx-y-z%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    Bressuire

                    Vorschmack

                    Quarantine