hadoop - Difference between Nutch crawl giving depth='N' and crawling in loop N times with depth='1' -


Background of my problem: I'm running Nutch1.4 at Hutchop0.20.203. I'm working on the niche segment to get the last output. But waiting for the entire crawl before running the matrade, the solution causes a long lasting run. Now I am triggering the mapping as soon as they are released, as soon as the jobs are removed from the areas. I give a depth = 1 to crawl a loop ('n = at the time of' depth). I am losing some URL while I crawl in the loop in depth 1, giving depth while making the N times equal.

Please get below the pseudo code:

Case 1 : Noah Crawl = 3

// on Hobachup Create a list object to store, which we are going to pass the Enchatch

list nutchArgsList = new ArrayList ();

nutchArgsList.add ("-depth");

nutchArgsList.add (integer.stosting (3));

& lt; ... other nutch args ... & gt; Case 2 : In Crawling In Loop 3, Crawling In Loop 3 (multiple times with depth) = 1 (int depth = Roe; =; Depth; Stretch; ;;; Depth; Run ++) {

// Create list items to store arguments that we are going to pass for nuts

list nutchArgsList = new ArrayList ();

nutchArgsList.add ("- Depth");

nutchArgsList.add (Integer.toString (1)); // Note I have provided depth as 1

& lt; ... other nutch args ... & gt;

ToolRunner.run (nutchConf, new crawl), nutchArgsList.toArray (new string [nutchArgsList.size ()]);

}

Getting url (DB footfade) When I crawling as a depth in the loop several times in the loop

I have tried it on standalone nach, where I walk 3 times vs 3 times the same with a 3x depth Runs on URL 1. I compare the difference between crawlbund and url only 12. But when I get 1000 urls from hoodop using the Touloner As far as DB_APFETED.

As far as I have understood, Nichtha triggers crawl in a loop as a depth value. Please suggest it.

Apart from this, please let me know that when I am using the tooler on the headpiece in the same way on different looks, then why is the difference?

Thanks in Adendas.

I have found that the behavior of nach alone (hard disk) Integrated directly with) running change and Hdop cluster. Generator score filtering appears to be much higher with a Hdop cluster, so "-topan" setting should be high enough.

I recommend running your crawl with a high (at least 1000) "TopN" and not

This is similar to my reaction.

After doing this, I came to know that my nails stand alone and HDFS started to start better. / P>

Comments

Post a Comment

Popular posts from this blog

Python SQLAlchemy：AttributeError: Neither 'Column' object nor 'Comparator' object has an attribute 'schema' -

I tried to create a new database in my project, but when I run the script, I get this error, I have another project, this type of definition worked before, but now it gets a single error. I am using Python 2.7.8 and the version of SQLAlchemy module is 0.9.8. By the way, using a project flask- SQLAlchemy, it works well, I'm confused. Traceback information is as follows: Traceback (most recent call final): File "D: / Projects / IOM / db_create.py", line 4, & lt; Module & gt; From the models the base file "D: \ Projects \ OO-IM \ models.py", in line 15, in the & lt; Module & gt; Column ( 'followed_id', int (), ForeignKey ( 'user.id')) file "C: \ Python27 \ lib \ site-packages \ SQLAlchemy \ SQL \ schema.py", line 369, __new__ schema = metadata. Schema file "C: \ Python27 \ lib \ site-packages \ SQLAlchemy \ SQL \ elements.py", line 662, AttributeError in __getattr__ key): Neither 'column' object nor ...
Read more

java - How not to audit a join table and related entities using Hibernate Envers? -

I use Eners to be hibernate to audit my institutions. I have an audit entity, Foo , which includes a list & lt; Bars & gt; However, I do not want to audit the bar institutions, in the form of properties, I have written that: @inti @edicated public class Foo {@JoinTable (name = "T_FOO_BAR", joinColumns = @JoinColumn (name = "FOO_ID"), inverseJoinColumns = @JoinColumn (Name = "BAR_ID")) @ManyToMany (Cascade = Persist) @ Indited (targetAuditMode = RelationTargetAuditMode.NOT_AUDITED ) Public listing & lt; Bars & gt; GetBars () {Return bars; }} Now, I want to retrieve an amendment of foo : AuditReader reader = AuditReaderFactory.get (GetEntityManager ()); FU modification = (FU) reader.CreativeTeA () .entensEtrevision (Foo Class, 42) .getSingleResult (); Unfortunately, when I want to retrieve all the data (i.e. when it loads lazy times ), error me ORA-00942: table or view is not present , because Tried to query this: Sel...
Read more

mongodb - CakePHP paginator ignoring order, but only for certain values -

While pagging my user model in kppp, I can sort with some values, but not others. For example, I can order results from created or email , for example, but username or reputation seems to come back order by arbitrary command for example with a list of users: $ this-> paginate = array ('condition' => array ( 'User.is_active' = & gt; true), 'border' => 24, 'order' => array ('User.created' => DESC)); Works as expected, but $ this - & gt; Paged = array ('conditions' = & gt; array (' user.isIactive '= & gt; true),' border '= & gt; 24,' command '= & gt; array (' user repeats' = ' & Gt; DESC ')); No, no. I first thought that this could be a database issue, but when I search the database directly, sorted as expected. Note: I am using Mangodebi with the Mongo DB plugin of Ichikawa for the KPPHP; Pagnetizing users are used to work properl...
Read more