multithreading - Process Large File for HTTP Calls in Java -


I have millions of lines on which I need a process. Each line in the file will result in an HTTP call. I am trying to figure out the best way to attack this problem.

I can clearly read the file and call it sequentially, but it will be incredibly slow. I want to parallelize the call, but I'm not sure if I should try to read the entire file in memory (something I'm not a big fan of) or (which I should try to parallelize the reading of the file simultaneously

The best way to attack this problem is to find some ideas here. If there is an existing structure or library that does something similar, then I am also happy to use it.

Thank you.

I want to parallelize calls, but I'm sure If I should not read the entire file in memory

You should use a executable service to use a surrounded blocking queue As you read in your million lines, you submit a job in the thread-pool until the block queue is complete. 100 will be able to run together simultaneously without having to read all of the lines of the HTTP request (or whatever number is optimal).

You will need to set up a DisclaimedAcquisitionHandler This block is at the completion of the queue. It is better than the collar run handler. 

  Blocking questions & lt; Runnabel & gt; Queue = new arrawlocking quatule & lt; Runnabel & gt; (100); // Note: You want to be the minimum and maximum thread number here in the same value ThreadPoolExecutor ThreadPool = new ThreadPoolExecutor (nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, queue); // If we need our RejectedExecutionHandler to block the thread Full threadPool.setRejectedExecutionHandler (New RejectedExecutionHandler {@Override Public} NulledExecution (Runnable R, ThreadPoolExecutor Executioner) {try // There is queue in the queue until this constructor Will block.Add executor.getQueue () (R);} hold (InterruptedException E) {New RejectedExecutionException ("Unexpected InterruptedException", e) Throwing;}}}); // Now when reading in URL ((string url = urlReader.readLine ())! = Null) {// submit them in thread-pool. This block may contain threads. Submit (new download URLRANABLE (URL)); } // After submitting, we must close the pool thread Pool.shutdown (); // Wait for them to complete Thread Paul. AwaitTermination (Long.MAX_VALUE, TimeUnit.MILLISECONDS); ... Private Category DownloadUrlRunnable Components Runnabel {Private Final string URL; Public DownloadUrlRunnable (string url) {this.url = url; } Public Zero Run () {// Download URL}}

Comments

Post a Comment

Popular posts from this blog

Python SQLAlchemy：AttributeError: Neither 'Column' object nor 'Comparator' object has an attribute 'schema' -

I tried to create a new database in my project, but when I run the script, I get this error, I have another project, this type of definition worked before, but now it gets a single error. I am using Python 2.7.8 and the version of SQLAlchemy module is 0.9.8. By the way, using a project flask- SQLAlchemy, it works well, I'm confused. Traceback information is as follows: Traceback (most recent call final): File "D: / Projects / IOM / db_create.py", line 4, & lt; Module & gt; From the models the base file "D: \ Projects \ OO-IM \ models.py", in line 15, in the & lt; Module & gt; Column ( 'followed_id', int (), ForeignKey ( 'user.id')) file "C: \ Python27 \ lib \ site-packages \ SQLAlchemy \ SQL \ schema.py", line 369, __new__ schema = metadata. Schema file "C: \ Python27 \ lib \ site-packages \ SQLAlchemy \ SQL \ elements.py", line 662, AttributeError in __getattr__ key): Neither 'column' object nor ...
Read more

java - How not to audit a join table and related entities using Hibernate Envers? -

I use Eners to be hibernate to audit my institutions. I have an audit entity, Foo , which includes a list & lt; Bars & gt; However, I do not want to audit the bar institutions, in the form of properties, I have written that: @inti @edicated public class Foo {@JoinTable (name = "T_FOO_BAR", joinColumns = @JoinColumn (name = "FOO_ID"), inverseJoinColumns = @JoinColumn (Name = "BAR_ID")) @ManyToMany (Cascade = Persist) @ Indited (targetAuditMode = RelationTargetAuditMode.NOT_AUDITED ) Public listing & lt; Bars & gt; GetBars () {Return bars; }} Now, I want to retrieve an amendment of foo : AuditReader reader = AuditReaderFactory.get (GetEntityManager ()); FU modification = (FU) reader.CreativeTeA () .entensEtrevision (Foo Class, 42) .getSingleResult (); Unfortunately, when I want to retrieve all the data (i.e. when it loads lazy times ), error me ORA-00942: table or view is not present , because Tried to query this: Sel...
Read more

mongodb - CakePHP paginator ignoring order, but only for certain values -

While pagging my user model in kppp, I can sort with some values, but not others. For example, I can order results from created or email , for example, but username or reputation seems to come back order by arbitrary command for example with a list of users: $ this-> paginate = array ('condition' => array ( 'User.is_active' = & gt; true), 'border' => 24, 'order' => array ('User.created' => DESC)); Works as expected, but $ this - & gt; Paged = array ('conditions' = & gt; array (' user.isIactive '= & gt; true),' border '= & gt; 24,' command '= & gt; array (' user repeats' = ' & Gt; DESC ')); No, no. I first thought that this could be a database issue, but when I search the database directly, sorted as expected. Note: I am using Mangodebi with the Mongo DB plugin of Ichikawa for the KPPHP; Pagnetizing users are used to work properl...
Read more