Java Multithread File Read + Operation + Write





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}







1















I am trying to write a small program to solve the following problems using multithreading in java. I am struggling to even understand where to start, and am looking for some advice. The desired steps in the process are as follows:




  1. Read in all the text files contained within a given directory

  2. Create a word count for each one of the files read.

  3. Write the count of words as an output to a new file in a different directory.


I have written the wordcount function, and that works fine, but would like to know more about how to multithread this operation so that the files are read, the words counted, and then the output is all written in parallel.






java multithreading







share|improve this question























  • Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

    – EdH
    Nov 16 '18 at 22:22




















1















I am trying to write a small program to solve the following problems using multithreading in java. I am struggling to even understand where to start, and am looking for some advice. The desired steps in the process are as follows:




  1. Read in all the text files contained within a given directory

  2. Create a word count for each one of the files read.

  3. Write the count of words as an output to a new file in a different directory.


I have written the wordcount function, and that works fine, but would like to know more about how to multithread this operation so that the files are read, the words counted, and then the output is all written in parallel.






java multithreading







share|improve this question























  • Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

    – EdH
    Nov 16 '18 at 22:22
















1












1








1








I am trying to write a small program to solve the following problems using multithreading in java. I am struggling to even understand where to start, and am looking for some advice. The desired steps in the process are as follows:




  1. Read in all the text files contained within a given directory

  2. Create a word count for each one of the files read.

  3. Write the count of words as an output to a new file in a different directory.


I have written the wordcount function, and that works fine, but would like to know more about how to multithread this operation so that the files are read, the words counted, and then the output is all written in parallel.






java multithreading







share|improve this question














I am trying to write a small program to solve the following problems using multithreading in java. I am struggling to even understand where to start, and am looking for some advice. The desired steps in the process are as follows:




  1. Read in all the text files contained within a given directory

  2. Create a word count for each one of the files read.

  3. Write the count of words as an output to a new file in a different directory.


I have written the wordcount function, and that works fine, but would like to know more about how to multithread this operation so that the files are read, the words counted, and then the output is all written in parallel.






java multithreading




java multithreading






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 16 '18 at 22:01









user10665129user10665129

61




61













  • Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

    – EdH
    Nov 16 '18 at 22:22





















  • Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

    – EdH
    Nov 16 '18 at 22:22



















Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

– EdH
Nov 16 '18 at 22:22







Perhaps using Stream functionality in Java (docs.oracle.com/javase/tutorial/collections/streams/…). Create a List of files contained within the directory, and get a parallelStream of those File objects. Then you can process each file, returning the word count as part of the parallel stream processing. Then you can figure out what to do with the count... if each is written to their own file, that too could be part of the stream processing. I'd start there

– EdH
Nov 16 '18 at 22:22














2 Answers
2






active

oldest

votes


















1














Can you share the single threaded version? Conceptually it can be as simple as this (pseudo Java code). countWords and writeOutput are your methods and
files is a list of files you already read in.



files.parallelStream()

    .map(file -> new Pair(file, countWords(file)))

    .forEach((file, count) -> writeOutput(file, count));





share|improve this answer































    0














    I like your enthusiastic in learning multi-threaded programming!

    Beside parallel your task between files, you can also parallel and optimize the actual countWords task.



    This is a classic case for using Fork/Join. A rather advance level of multi-threaded programming but a very satisfying one :)



    you can search the web for examples similar to your own, here is a one to start with: https://www.baeldung.com/java-fork-join






    share|improve this answer
























      Your Answer






      StackExchange.ifUsing("editor", function () {
      StackExchange.using("externalEditor", function () {
      StackExchange.using("snippets", function () {
      StackExchange.snippets.init();
      });
      });
      }, "code-snippets");

      StackExchange.ready(function() {
      var channelOptions = {
      tags: "".split(" "),
      id: "1"
      };
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function() {
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled) {
      StackExchange.using("snippets", function() {
      createEditor();
      });
      }
      else {
      createEditor();
      }
      });

      function createEditor() {
      StackExchange.prepareEditor({
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: true,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: 10,
      bindNavPrevention: true,
      postfix: "",
      imageUploader: {
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      },
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      });


      }
      });














      draft saved

      draft discarded


















      StackExchange.ready(
      function () {
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53346013%2fjava-multithread-file-read-operation-write%23new-answer', 'question_page');
      }
      );

      Post as a guest















      Required, but never shown

























      2 Answers
      2






      active

      oldest

      votes








      2 Answers
      2






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      1














      Can you share the single threaded version? Conceptually it can be as simple as this (pseudo Java code). countWords and writeOutput are your methods and
      files is a list of files you already read in.



      files.parallelStream()
      
    .map(file -> new Pair(file, countWords(file)))
      
    .forEach((file, count) -> writeOutput(file, count));





      share|improve this answer




























        1














        Can you share the single threaded version? Conceptually it can be as simple as this (pseudo Java code). countWords and writeOutput are your methods and
        files is a list of files you already read in.



        files.parallelStream()
        
    .map(file -> new Pair(file, countWords(file)))
        
    .forEach((file, count) -> writeOutput(file, count));





        share|improve this answer


























          1












          1








          1







          Can you share the single threaded version? Conceptually it can be as simple as this (pseudo Java code). countWords and writeOutput are your methods and
          files is a list of files you already read in.



          files.parallelStream()
          
    .map(file -> new Pair(file, countWords(file)))
          
    .forEach((file, count) -> writeOutput(file, count));





          share|improve this answer













          Can you share the single threaded version? Conceptually it can be as simple as this (pseudo Java code). countWords and writeOutput are your methods and
          files is a list of files you already read in.



          files.parallelStream()
          
    .map(file -> new Pair(file, countWords(file)))
          
    .forEach((file, count) -> writeOutput(file, count));






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 16 '18 at 22:24









          K. M. Fazle Azim BabuK. M. Fazle Azim Babu

          1264




          1264

























              0














              I like your enthusiastic in learning multi-threaded programming!

              Beside parallel your task between files, you can also parallel and optimize the actual countWords task.



              This is a classic case for using Fork/Join. A rather advance level of multi-threaded programming but a very satisfying one :)



              you can search the web for examples similar to your own, here is a one to start with: https://www.baeldung.com/java-fork-join






              share|improve this answer




























                0














                I like your enthusiastic in learning multi-threaded programming!

                Beside parallel your task between files, you can also parallel and optimize the actual countWords task.



                This is a classic case for using Fork/Join. A rather advance level of multi-threaded programming but a very satisfying one :)



                you can search the web for examples similar to your own, here is a one to start with: https://www.baeldung.com/java-fork-join






                share|improve this answer


























                  0












                  0








                  0







                  I like your enthusiastic in learning multi-threaded programming!

                  Beside parallel your task between files, you can also parallel and optimize the actual countWords task.



                  This is a classic case for using Fork/Join. A rather advance level of multi-threaded programming but a very satisfying one :)



                  you can search the web for examples similar to your own, here is a one to start with: https://www.baeldung.com/java-fork-join






                  share|improve this answer













                  I like your enthusiastic in learning multi-threaded programming!

                  Beside parallel your task between files, you can also parallel and optimize the actual countWords task.



                  This is a classic case for using Fork/Join. A rather advance level of multi-threaded programming but a very satisfying one :)



                  you can search the web for examples similar to your own, here is a one to start with: https://www.baeldung.com/java-fork-join







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered Nov 16 '18 at 23:13









                  doronydorony

                  363213




                  363213






























                      draft saved

                      draft discarded




















































                      Thanks for contributing an answer to Stack Overflow!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid …



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function () {
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53346013%2fjava-multithread-file-read-operation-write%23new-answer', 'question_page');
                      }
                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      -

                      Popular posts from this blog

                      Bressuire

                      Image
                      Subprefecture and commune in Nouvelle-Aquitaine, France Bressuire Subprefecture and commune Chateau de Bressuire and the Eglise Notre-Dame Coat of arms Location of Bressuire Bressuire Show map of France Bressuire Show map of Nouvelle-Aquitaine Coordinates: 46°50′27″N 0°29′14″W  /  46.8408°N 0.4872°W  / 46.8408; -0.4872 Coordinates: 46°50′27″N 0°29′14″W  /  46.8408°N 0.4872°W  / 46.8408; -0.4872 Country France Region Nouvelle-Aquitaine Department Deux-Sèvres Arrondissement Bressuire Canton Bressuire Government  • Mayor .mw-parser-output .nobold{font-weight:normal} (2014–20) Jean Michel Bernier Area 1 180.59 km 2 (69.73 sq mi) Population (2014) 2 19,300  • Density 110/km 2 (280/sq mi) Time zone UTC+01:00 (CET)  • Summer (DST) UTC+02:00 (CEST) INSEE/Postal code 79049 /79300 Elevation 98–236 m (322–774 ft) (avg. 173 m or 568 ft) 1 French Land Register data, which exclude...
                      Read more

                      Vorschmack

                      Image
                      Vorschmack Ukrainian Jewish-style vorschmack served on rye bread Course Hors d'oeuvre Region or state Eastern Europe Associated national cuisine Ashkenazi Jewish, Finnish, German, Ukrainian, Polish, Russian Main ingredients Ground meat and/or fish Cookbook: Vorschmack   Media: Vorschmack Vorschmack or forshmak (Yiddish: פֿאָרשמאַק ‎, from archaic German Vorschmack , "foretaste" [1] or "appetizer" [2] ) is an originally East European dish made of salty minced fish or meat. Different variants of this dish are especially common in Ashkenazi Jewish and Finnish cuisine. Some varieties are also known in Russian and Polish cuisine. Contents 1 In Jewish cuisine 2 In Russian cuisine 3 In Polish cuisine 4 In Finnish cuisine 5 See also 6 References In Jewish cuisine According to Gil Marks, the German name points to the possible Germanic origin of this dish. [1] William Pokhlyobkin descr...
                      Read more

                      Quarantine

                      Image
                      For other uses, see Quarantine (disambiguation). Signal flag "Lima" called the "Yellow Jack" which when flown in harbor means ship is under quarantine. A simple yellow flag (also called the "Yellow Jack") had historically been used to signal quarantine (it stands for Q among signal flags), but now indicates the opposite, as a signal of a ship free of disease that requests boarding and inspection. A quarantine is used to separate and restrict the movement of people; it is 'a restraint upon the activities or communication of persons or the transport of goods designed to prevent the spread of disease or pests', for a certain period of time. [1] This is often used in connection to disease and illness, such as those who may possibly have been exposed to a communicable disease. [2] The term is often erroneously used to mean medical isolation, which is "to separate ill persons who have a communicable disease from those who are healthy...
                      Read more