How to COPY library files between stages of a multi-stage Docker build while preserving symlinks?












1















I have a Dockerfile which is split into a two-stage multi-stage docker build. The first stage generates a basic gcc build environment in which a number of C and C++ library are compiled. The second stage uses the COPY --from= command to copy the library files from the first stages /usr/local/lib/libproto* to the current image's.



The problem I am seeing is that the first image contains symlinks from a generic library file name to a specific versioned file name. AFAIK this is common practice within Debian and many other Linux systems. Docker's COPY command does not seem to understand symlinks so instead makes two complete copies of the library files. This results in a larger Docker Image size and warnings from later apt-get commands to the tune of ldconfig: /usr/local/lib/libprotobuf.so.17 is not a symbolic link.





My specific file presently looks like:



#Compile any tools we cannot install from packages
FROM gcc:7 as builder
USER 0
RUN
apt-get -y update &&
apt-get -y install
clang
libc++-dev
libgflags-dev
libgtest-dev
RUN
# Protocol Buffer & gRPC
# install protobuf first, then grpc
git clone -b $(curl -L https://grpc.io/release)
https://github.com/grpc/grpc /var/local/git/grpc &&
cd /var/local/git/grpc &&
git submodule update --init &&
echo "--- installing protobuf ---" &&
cd third_party/protobuf &&
./autogen.sh && ./configure --enable-shared &&
make -j$(nproc) && make install && make clean && ldconfig &&
echo "--- installing grpc ---" &&
cd /var/local/git/grpc &&
make -j$(nproc) && make install && make clean && ldconfig


FROM debian
LABEL
Description="Basic Debian production environment with a number of libraries configured"
MAINTAINER="Mr Me"
ARG prefix=/usr/local
ARG binPath=$prefix/bin
ARG libPath=$prefix/lib
# Copy over pre-made tools
# Protocol Buffer
COPY --from=builder /usr/local/lib/libproto* $libPath/
# gRPC
COPY --from=builder /usr/local/lib/libaddress_sorting.so.6.0.0 $libPath/
COPY --from=builder /usr/local/lib/libgpr* $libPath/
COPY --from=builder /usr/local/lib/libgrpc* $libPath/
RUN ldconfig
# Install remaining tools using apt-get
RUN apt-get -y update &&
apt-get -y install
libhdf5-dev
libssl1.1
uuid-dev;


As you can see I am trying to add the latest versions of gRPC and Protocol Buffer to a Debian based runtime image.










share|improve this question























  • As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.

    – Jacob Tomlinson
    Nov 13 '18 at 15:46











  • @JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?

    – TafT
    Nov 13 '18 at 16:42











  • It is so clunky I didn't feel like I should XD. I will do now at your request.

    – Jacob Tomlinson
    Nov 13 '18 at 16:45
















1















I have a Dockerfile which is split into a two-stage multi-stage docker build. The first stage generates a basic gcc build environment in which a number of C and C++ library are compiled. The second stage uses the COPY --from= command to copy the library files from the first stages /usr/local/lib/libproto* to the current image's.



The problem I am seeing is that the first image contains symlinks from a generic library file name to a specific versioned file name. AFAIK this is common practice within Debian and many other Linux systems. Docker's COPY command does not seem to understand symlinks so instead makes two complete copies of the library files. This results in a larger Docker Image size and warnings from later apt-get commands to the tune of ldconfig: /usr/local/lib/libprotobuf.so.17 is not a symbolic link.





My specific file presently looks like:



#Compile any tools we cannot install from packages
FROM gcc:7 as builder
USER 0
RUN
apt-get -y update &&
apt-get -y install
clang
libc++-dev
libgflags-dev
libgtest-dev
RUN
# Protocol Buffer & gRPC
# install protobuf first, then grpc
git clone -b $(curl -L https://grpc.io/release)
https://github.com/grpc/grpc /var/local/git/grpc &&
cd /var/local/git/grpc &&
git submodule update --init &&
echo "--- installing protobuf ---" &&
cd third_party/protobuf &&
./autogen.sh && ./configure --enable-shared &&
make -j$(nproc) && make install && make clean && ldconfig &&
echo "--- installing grpc ---" &&
cd /var/local/git/grpc &&
make -j$(nproc) && make install && make clean && ldconfig


FROM debian
LABEL
Description="Basic Debian production environment with a number of libraries configured"
MAINTAINER="Mr Me"
ARG prefix=/usr/local
ARG binPath=$prefix/bin
ARG libPath=$prefix/lib
# Copy over pre-made tools
# Protocol Buffer
COPY --from=builder /usr/local/lib/libproto* $libPath/
# gRPC
COPY --from=builder /usr/local/lib/libaddress_sorting.so.6.0.0 $libPath/
COPY --from=builder /usr/local/lib/libgpr* $libPath/
COPY --from=builder /usr/local/lib/libgrpc* $libPath/
RUN ldconfig
# Install remaining tools using apt-get
RUN apt-get -y update &&
apt-get -y install
libhdf5-dev
libssl1.1
uuid-dev;


As you can see I am trying to add the latest versions of gRPC and Protocol Buffer to a Debian based runtime image.










share|improve this question























  • As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.

    – Jacob Tomlinson
    Nov 13 '18 at 15:46











  • @JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?

    – TafT
    Nov 13 '18 at 16:42











  • It is so clunky I didn't feel like I should XD. I will do now at your request.

    – Jacob Tomlinson
    Nov 13 '18 at 16:45














1












1








1








I have a Dockerfile which is split into a two-stage multi-stage docker build. The first stage generates a basic gcc build environment in which a number of C and C++ library are compiled. The second stage uses the COPY --from= command to copy the library files from the first stages /usr/local/lib/libproto* to the current image's.



The problem I am seeing is that the first image contains symlinks from a generic library file name to a specific versioned file name. AFAIK this is common practice within Debian and many other Linux systems. Docker's COPY command does not seem to understand symlinks so instead makes two complete copies of the library files. This results in a larger Docker Image size and warnings from later apt-get commands to the tune of ldconfig: /usr/local/lib/libprotobuf.so.17 is not a symbolic link.





My specific file presently looks like:



#Compile any tools we cannot install from packages
FROM gcc:7 as builder
USER 0
RUN
apt-get -y update &&
apt-get -y install
clang
libc++-dev
libgflags-dev
libgtest-dev
RUN
# Protocol Buffer & gRPC
# install protobuf first, then grpc
git clone -b $(curl -L https://grpc.io/release)
https://github.com/grpc/grpc /var/local/git/grpc &&
cd /var/local/git/grpc &&
git submodule update --init &&
echo "--- installing protobuf ---" &&
cd third_party/protobuf &&
./autogen.sh && ./configure --enable-shared &&
make -j$(nproc) && make install && make clean && ldconfig &&
echo "--- installing grpc ---" &&
cd /var/local/git/grpc &&
make -j$(nproc) && make install && make clean && ldconfig


FROM debian
LABEL
Description="Basic Debian production environment with a number of libraries configured"
MAINTAINER="Mr Me"
ARG prefix=/usr/local
ARG binPath=$prefix/bin
ARG libPath=$prefix/lib
# Copy over pre-made tools
# Protocol Buffer
COPY --from=builder /usr/local/lib/libproto* $libPath/
# gRPC
COPY --from=builder /usr/local/lib/libaddress_sorting.so.6.0.0 $libPath/
COPY --from=builder /usr/local/lib/libgpr* $libPath/
COPY --from=builder /usr/local/lib/libgrpc* $libPath/
RUN ldconfig
# Install remaining tools using apt-get
RUN apt-get -y update &&
apt-get -y install
libhdf5-dev
libssl1.1
uuid-dev;


As you can see I am trying to add the latest versions of gRPC and Protocol Buffer to a Debian based runtime image.










share|improve this question














I have a Dockerfile which is split into a two-stage multi-stage docker build. The first stage generates a basic gcc build environment in which a number of C and C++ library are compiled. The second stage uses the COPY --from= command to copy the library files from the first stages /usr/local/lib/libproto* to the current image's.



The problem I am seeing is that the first image contains symlinks from a generic library file name to a specific versioned file name. AFAIK this is common practice within Debian and many other Linux systems. Docker's COPY command does not seem to understand symlinks so instead makes two complete copies of the library files. This results in a larger Docker Image size and warnings from later apt-get commands to the tune of ldconfig: /usr/local/lib/libprotobuf.so.17 is not a symbolic link.





My specific file presently looks like:



#Compile any tools we cannot install from packages
FROM gcc:7 as builder
USER 0
RUN
apt-get -y update &&
apt-get -y install
clang
libc++-dev
libgflags-dev
libgtest-dev
RUN
# Protocol Buffer & gRPC
# install protobuf first, then grpc
git clone -b $(curl -L https://grpc.io/release)
https://github.com/grpc/grpc /var/local/git/grpc &&
cd /var/local/git/grpc &&
git submodule update --init &&
echo "--- installing protobuf ---" &&
cd third_party/protobuf &&
./autogen.sh && ./configure --enable-shared &&
make -j$(nproc) && make install && make clean && ldconfig &&
echo "--- installing grpc ---" &&
cd /var/local/git/grpc &&
make -j$(nproc) && make install && make clean && ldconfig


FROM debian
LABEL
Description="Basic Debian production environment with a number of libraries configured"
MAINTAINER="Mr Me"
ARG prefix=/usr/local
ARG binPath=$prefix/bin
ARG libPath=$prefix/lib
# Copy over pre-made tools
# Protocol Buffer
COPY --from=builder /usr/local/lib/libproto* $libPath/
# gRPC
COPY --from=builder /usr/local/lib/libaddress_sorting.so.6.0.0 $libPath/
COPY --from=builder /usr/local/lib/libgpr* $libPath/
COPY --from=builder /usr/local/lib/libgrpc* $libPath/
RUN ldconfig
# Install remaining tools using apt-get
RUN apt-get -y update &&
apt-get -y install
libhdf5-dev
libssl1.1
uuid-dev;


As you can see I am trying to add the latest versions of gRPC and Protocol Buffer to a Debian based runtime image.







c++ docker shared-libraries dockerfile docker-multi-stage-build






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 13 '18 at 13:00









TafTTafT

7471923




7471923













  • As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.

    – Jacob Tomlinson
    Nov 13 '18 at 15:46











  • @JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?

    – TafT
    Nov 13 '18 at 16:42











  • It is so clunky I didn't feel like I should XD. I will do now at your request.

    – Jacob Tomlinson
    Nov 13 '18 at 16:45



















  • As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.

    – Jacob Tomlinson
    Nov 13 '18 at 15:46











  • @JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?

    – TafT
    Nov 13 '18 at 16:42











  • It is so clunky I didn't feel like I should XD. I will do now at your request.

    – Jacob Tomlinson
    Nov 13 '18 at 16:45

















As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.

– Jacob Tomlinson
Nov 13 '18 at 15:46





As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.

– Jacob Tomlinson
Nov 13 '18 at 15:46













@JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?

– TafT
Nov 13 '18 at 16:42





@JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?

– TafT
Nov 13 '18 at 16:42













It is so clunky I didn't feel like I should XD. I will do now at your request.

– Jacob Tomlinson
Nov 13 '18 at 16:45





It is so clunky I didn't feel like I should XD. I will do now at your request.

– Jacob Tomlinson
Nov 13 '18 at 16:45












1 Answer
1






active

oldest

votes


















1














This is more of a workaround than an answer.



You could tar the files, copy the tarball to the second container and then untar them.



Tar maintains symbolic links by default.






share|improve this answer























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53281570%2fhow-to-copy-library-files-between-stages-of-a-multi-stage-docker-build-while-pre%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    1














    This is more of a workaround than an answer.



    You could tar the files, copy the tarball to the second container and then untar them.



    Tar maintains symbolic links by default.






    share|improve this answer




























      1














      This is more of a workaround than an answer.



      You could tar the files, copy the tarball to the second container and then untar them.



      Tar maintains symbolic links by default.






      share|improve this answer


























        1












        1








        1







        This is more of a workaround than an answer.



        You could tar the files, copy the tarball to the second container and then untar them.



        Tar maintains symbolic links by default.






        share|improve this answer













        This is more of a workaround than an answer.



        You could tar the files, copy the tarball to the second container and then untar them.



        Tar maintains symbolic links by default.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 13 '18 at 16:48









        Jacob TomlinsonJacob Tomlinson

        1,60221946




        1,60221946






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53281570%2fhow-to-copy-library-files-between-stages-of-a-multi-stage-docker-build-while-pre%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Xamarin.iOS Cant Deploy on Iphone

            Glorious Revolution

            Dulmage-Mendelsohn matrix decomposition in Python