How to bundle Python packages (case study: Microsoft Visual Studio 2017)
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I am building Python from source, the advantage of this being it will be optimized for my hardware (PGO). But I also want to compile 3rd party packages like NumPy or SciPy to take advantage of intel MKL (I have a Xeon CPU) and optimize them instead of just installing them via PIP which would download pre-compiled, "generic", non-optimized code.
Because I would potentially compile "a lot" of packages by myself I am looking for suggestions as to how to automatize this process. The "traditional" or "standard" way of doing this would be something like this:
1) compile a package
2) include that package to python (maybe using PIP?)
3) repeat the previous 2 steps for each package.
I intend to use Microsoft Visual Studio (under Windows 10 x64), so my question is this:
Is there a way to automatize this process using Visual Studio? if so,
Can anyone give me a hint as to what to look for?
I am new to Visual Studio and I am not asking for a complete "How To" but I would appreciate someone telling me if this can be done (or feasible, depending on what packages need to be compiled), otherwise I would revert back to manually make that process by compiling and adding them to Python one by one.
python numpy build compilation scipy
asked Nov 16 '18 at 17:41
J. RamosJ. Ramos
236
add a comment |
I am building Python from source, the advantage of this being it will be optimized for my hardware (PGO). But I also want to compile 3rd party packages like NumPy or SciPy to take advantage of intel MKL (I have a Xeon CPU) and optimize them instead of just installing them via PIP which would download pre-compiled, "generic", non-optimized code.
Because I would potentially compile "a lot" of packages by myself I am looking for suggestions as to how to automatize this process. The "traditional" or "standard" way of doing this would be something like this:
1) compile a package
2) include that package to python (maybe using PIP?)
3) repeat the previous 2 steps for each package.
I intend to use Microsoft Visual Studio (under Windows 10 x64), so my question is this:
Is there a way to automatize this process using Visual Studio? if so,
Can anyone give me a hint as to what to look for?
I am new to Visual Studio and I am not asking for a complete "How To" but I would appreciate someone telling me if this can be done (or feasible, depending on what packages need to be compiled), otherwise I would revert back to manually make that process by compiling and adding them to Python one by one.
python numpy build compilation scipy
asked Nov 16 '18 at 17:41
J. RamosJ. Ramos
236
isnt that justmake?
– Joran Beasley
Nov 16 '18 at 17:47
I think it would be easier to use Anaconda Python, which already has precompiled numpy, scipy with Intel MKL or get the packages from lfd.uci.edu/~gohlke/pythonlibs (also with MKL)
– max9111
Nov 19 '18 at 7:59
add a comment |
I am building Python from source, the advantage of this being it will be optimized for my hardware (PGO). But I also want to compile 3rd party packages like NumPy or SciPy to take advantage of intel MKL (I have a Xeon CPU) and optimize them instead of just installing them via PIP which would download pre-compiled, "generic", non-optimized code.
Because I would potentially compile "a lot" of packages by myself I am looking for suggestions as to how to automatize this process. The "traditional" or "standard" way of doing this would be something like this:
1) compile a package
2) include that package to python (maybe using PIP?)
3) repeat the previous 2 steps for each package.
I intend to use Microsoft Visual Studio (under Windows 10 x64), so my question is this:
Is there a way to automatize this process using Visual Studio? if so,
Can anyone give me a hint as to what to look for?
I am new to Visual Studio and I am not asking for a complete "How To" but I would appreciate someone telling me if this can be done (or feasible, depending on what packages need to be compiled), otherwise I would revert back to manually make that process by compiling and adding them to Python one by one.
python numpy build compilation scipy
asked Nov 16 '18 at 17:41
J. RamosJ. Ramos
236
I am building Python from source, the advantage of this being it will be optimized for my hardware (PGO). But I also want to compile 3rd party packages like NumPy or SciPy to take advantage of intel MKL (I have a Xeon CPU) and optimize them instead of just installing them via PIP which would download pre-compiled, "generic", non-optimized code.
Because I would potentially compile "a lot" of packages by myself I am looking for suggestions as to how to automatize this process. The "traditional" or "standard" way of doing this would be something like this:
1) compile a package
2) include that package to python (maybe using PIP?)
3) repeat the previous 2 steps for each package.
I intend to use Microsoft Visual Studio (under Windows 10 x64), so my question is this:
Is there a way to automatize this process using Visual Studio? if so,
Can anyone give me a hint as to what to look for?
I am new to Visual Studio and I am not asking for a complete "How To" but I would appreciate someone telling me if this can be done (or feasible, depending on what packages need to be compiled), otherwise I would revert back to manually make that process by compiling and adding them to Python one by one.
python numpy build compilation scipy
python numpy build compilation scipy
asked Nov 16 '18 at 17:41
J. RamosJ. Ramos
236
asked Nov 16 '18 at 17:41
J. RamosJ. Ramos
236
asked Nov 16 '18 at 17:41
J. RamosJ. Ramos
236
asked Nov 16 '18 at 17:41
J. RamosJ. Ramos
236
asked Nov 16 '18 at 17:41
J. RamosJ. Ramos
236
236
isnt that justmake?
– Joran Beasley
Nov 16 '18 at 17:47
I think it would be easier to use Anaconda Python, which already has precompiled numpy, scipy with Intel MKL or get the packages from lfd.uci.edu/~gohlke/pythonlibs (also with MKL)
– max9111
Nov 19 '18 at 7:59
add a comment |
isnt that justmake?
– Joran Beasley
Nov 16 '18 at 17:47
I think it would be easier to use Anaconda Python, which already has precompiled numpy, scipy with Intel MKL or get the packages from lfd.uci.edu/~gohlke/pythonlibs (also with MKL)
– max9111
Nov 19 '18 at 7:59
isnt that just
make?– Joran Beasley
Nov 16 '18 at 17:47
isnt that just
make?– Joran Beasley
Nov 16 '18 at 17:47
I think it would be easier to use Anaconda Python, which already has precompiled numpy, scipy with Intel MKL or get the packages from lfd.uci.edu/~gohlke/pythonlibs (also with MKL)
– max9111
Nov 19 '18 at 7:59
I think it would be easier to use Anaconda Python, which already has precompiled numpy, scipy with Intel MKL or get the packages from lfd.uci.edu/~gohlke/pythonlibs (also with MKL)
– max9111
Nov 19 '18 at 7:59
add a comment |
1 Answer
1
active
oldest
votes
If you're looking to optimize, you can't really automate much as each library will need its own customization. otherwise pip is probably the way to go. Realistically however, you'll only probably need to optimize a few libraries such as the scientific stack and possibly numba for better jit compilation.
TLDR; install numpy from source using the blas/lapack (math libs) optimized for your system and use pip for the rest.
answered Nov 16 '18 at 17:54
AaronAaron
4,70611427
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53342871%2fhow-to-bundle-python-packages-case-study-microsoft-visual-studio-2017%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If you're looking to optimize, you can't really automate much as each library will need its own customization. otherwise pip is probably the way to go. Realistically however, you'll only probably need to optimize a few libraries such as the scientific stack and possibly numba for better jit compilation.
TLDR; install numpy from source using the blas/lapack (math libs) optimized for your system and use pip for the rest.
answered Nov 16 '18 at 17:54
AaronAaron
4,70611427
add a comment |
If you're looking to optimize, you can't really automate much as each library will need its own customization. otherwise pip is probably the way to go. Realistically however, you'll only probably need to optimize a few libraries such as the scientific stack and possibly numba for better jit compilation.
TLDR; install numpy from source using the blas/lapack (math libs) optimized for your system and use pip for the rest.
answered Nov 16 '18 at 17:54
AaronAaron
4,70611427
add a comment |
If you're looking to optimize, you can't really automate much as each library will need its own customization. otherwise pip is probably the way to go. Realistically however, you'll only probably need to optimize a few libraries such as the scientific stack and possibly numba for better jit compilation.
TLDR; install numpy from source using the blas/lapack (math libs) optimized for your system and use pip for the rest.
answered Nov 16 '18 at 17:54
AaronAaron
4,70611427
If you're looking to optimize, you can't really automate much as each library will need its own customization. otherwise pip is probably the way to go. Realistically however, you'll only probably need to optimize a few libraries such as the scientific stack and possibly numba for better jit compilation.
TLDR; install numpy from source using the blas/lapack (math libs) optimized for your system and use pip for the rest.
answered Nov 16 '18 at 17:54
AaronAaron
4,70611427
answered Nov 16 '18 at 17:54
AaronAaron
4,70611427
answered Nov 16 '18 at 17:54
AaronAaron
4,70611427
answered Nov 16 '18 at 17:54
AaronAaron
4,70611427
4,70611427
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53342871%2fhow-to-bundle-python-packages-case-study-microsoft-visual-studio-2017%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
isnt that just
make?– Joran Beasley
Nov 16 '18 at 17:47
I think it would be easier to use Anaconda Python, which already has precompiled numpy, scipy with Intel MKL or get the packages from lfd.uci.edu/~gohlke/pythonlibs (also with MKL)
– max9111
Nov 19 '18 at 7:59