- Where Developers Learn, Share, & Build Careers
I have millions of lines on which I need a process. Each line in the file will result in an HTTP call. I am trying to figure out the best way to attack this problem.
I can clearly read the file and call it sequentially, but it will be incredibly slow. I would like to call parallelize, but I'm not sure if I am not a big fan of the whole file into memory (something) I am reading (or who do not want me to try as well to parallelize read the file
The best way to attack this problem is to find some ideas here. If there is an existing structure or library that does something similar, then I am also happy to use it.
Thank you.
I would like to call parallelize, but I'm sure If I'm not supposed to read the entire file into memory
you must have a
executor Services using an enclosedblock queue As you read in your million lines, you submit a job in the thread-pool until theblock queue is complete. 100 will be able to run together simultaneously without having to read all of the lines of the HTTP request (or whatever number is optimal).You will need to set up a
DisclaimedAcquisitionHandler This block is at the completion of the queue. It is better than the collar run handler.Blocking questions & lt; Runnabel & gt; Queue = new arrawlocking quatule & lt; Runnabel & gt; (100); // Note: you want to be the minimum and maximum on the thread count the same value ThreadPoolExecutor ThreadPool = new ThreadPoolExecutor (nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, Qatar); // If we queue full threadPool.setRejectedExecutionHandler is needed our RejectedExecutionHandler to block (New RejectedExecutionHandler () {@Override public void rejectedExecution (Runnable r, ThreadPoolExecutor executor) {try {there is room in the queue until // this manufacturer put. executor.getQueue () (R) will block;} throw catch (InterruptedException e) {new RejectedExecutionException ( "unexpected InterruptedException", e);}}}); // Now when reading in URL ((string url = urlReader.readLine ())! = Null) {// submit them in thread-pool. This block may contain threads. Submit (new download URLRANABLE (URL)); } // After submitting, we must close the pool thread Pool.shutdown (); // Wait for them to complete Thread Paul. AwaitTermination (Long.MAX_VALUE, TimeUnit.MILLISECONDS); ... Private Category DownloadUrlRunnable Components Runnabel {Private Final string URL; Public DownloadUrlRunnable (string url) {this.url = url; } Public Zero Run () {// Download URL}}
Comments
Post a Comment