php - Even CURL function can't scrape some urls -


I am using curl to scrap the html of the url. It uses 80% URLs which I use. But some URLs do not seem to "scrape", for example, when I try to scrape, it does not work. The website is loading and in the end it does not return any results. The problem is that this is my code: 

  if ($ _ POST ['submit']) {function curl_exec_follow ($ ch , & Max $ max = null) {$ mr = $ Maxredirect === Tap? 5: difference ($ max); If (ini_get ( 'open_basedir') == '' & amp; amp; ini_get ( 'safe_mode' == 'off')) {curl_setopt ($ ch, CURLOPT_FOLLOWLOCATION, $ mr & gt; 0); Curl_setopt ($ CH, CURLOPT_MAXREDIRS, $ MR); Curl_setopt ($ CH, CURLOPT_RETURNTRANSFER, true); Curl_setopt ($ CH, CURLOPT_SSL_VERIFYPEER, incorrect); } And (curl_setopt ($ ch, CURLOPT_FOLLOWLOCATION, false); if ($ mr & gt; 0) {$ original_url = curl_getinfo ($ ch, CURLINFO_EFFECTIVE_URL); $ Newurl = $ original_url; $ Rch = curl_copy_handle ($ ch); Curl_setopt ($ rch, CURLOPT_HEADER, true); Curl_setopt ($ rch, CURLOPT_NOBODY, true); Curl_setopt ($ rch, CURLOPT_FORBID_REUSE, false); to {curl_setopt ($ rch, CURLOPT_URL, $ newurl); $ header = Krl_akssi ($ RC) ; if (curl_errno ($ rch)) {$ code = 0;} and {$ code = curl_getinfo ($ rch, CURLINFO_HTTP_CODE); if ($ code == 301 || $ code == 302) {preg_match ( '/ location : $. * $?); $ Newurl = trim (array_pop ($ matches)); // If no scheme exists then the new URL is a // relative path and thus in Additionally Need Ekbal if (! Preg_match ( "/ ^ https https: / i", $ newurl)) {$ newurl = $ original_url $ NEWURL;}} and {$ code = 0;}}} while ($ code & amp ; - $ mr); curl_close ($ RCH); if (! $ MR) {if ($ maxredirect === null) trigger_upper ('too much redirection.', E_USER_WARNING); Else $ maxredirect = 0; return false; } Curl_setopt ($ ch, CURLOPT_URL, $ newurl); }} Return curl_exec ($ ch); } $ Ch = curl_init ($ _ POST ['form_url']); Curl_setopt ($ CH, CURLOPT_RETURNTRANSFER, true); $ Data = curl_xax_fol ($ f); Curl_close ($ ch); $ Data echo;    
 
 
 try it .. hopefully it helps ...  
   Curl, CURLOPT_HTTPHEADER, $ header); Curl_setopt ($ this-> Curl, CURLOPT_COOKIEJAR, $ cookiejar); Curl_setopt ($ this-> Curl, CURLOPT_COOKIEFILE, $ cookiejar); Curl_setopt ($ this-> Curl, CURLOPT_AUTOREFERER, true); Curl_setopt ($ this-> Curl, CURLOPT_FOLLOWLOCATION, true); Curl_setopt ($ this-> Curl, CURLOPT_RETURNTRANSFER, is true); } Find function ($ url) {$ this- & gt; Curl = curl_init ($ url); $ This- & gt; Set-up (); Return $ $-> request (); } GetAll function ($ reg, $ str) {preg_match_all ($ reg, $ str, $ matches); Returns $ matches [1]; } Festival postForm ($ url, $ fields, $ referer = '') {$ this- & gt; Curl = curl_init ($ url); $ This- & gt; Set-up (); Curl_setopt ($ this-> Curl, CURLOPT_URL, $ url); Curl_setopt ($ this-> Curl, CURLOPT_POST, 1); Curl_setopt ($ this-> Curl, CURLOPT_REFERER, $ referer); Curl_setopt ($ this- & gt; curl, CURLOPT_POSTFIELDS, $ fields); Return $ $-> request (); } Function getInfo ($ info) {$ info = ($ info == 'last')? Curl_getinfo ($ this-> Curl, CURLINFO_EFFECTIVE_URL): curl_getinfo ($ this-> Curl, $ info); Return $ info; } Function request () {return curl_exec ($ this-> curl); }} {$ Curl = new curls (); $ Html = $ curl- & gt; Get ("http://www.thefancy.com"); Echo "$ html"; }? & Gt;

Comments

Post a Comment

Popular posts from this blog

Python SQLAlchemy：AttributeError: Neither 'Column' object nor 'Comparator' object has an attribute 'schema' -

I tried to create a new database in my project, but when I run the script, I get this error, I have another project, this type of definition worked before, but now it gets a single error. I am using Python 2.7.8 and the version of SQLAlchemy module is 0.9.8. By the way, using a project flask- SQLAlchemy, it works well, I'm confused. Traceback information is as follows: Traceback (most recent call final): File "D: / Projects / IOM / db_create.py", line 4, & lt; Module & gt; From the models the base file "D: \ Projects \ OO-IM \ models.py", in line 15, in the & lt; Module & gt; Column ( 'followed_id', int (), ForeignKey ( 'user.id')) file "C: \ Python27 \ lib \ site-packages \ SQLAlchemy \ SQL \ schema.py", line 369, __new__ schema = metadata. Schema file "C: \ Python27 \ lib \ site-packages \ SQLAlchemy \ SQL \ elements.py", line 662, AttributeError in __getattr__ key): Neither 'column' object nor ...
Read more

java - How not to audit a join table and related entities using Hibernate Envers? -

I use Eners to be hibernate to audit my institutions. I have an audit entity, Foo , which includes a list & lt; Bars & gt; However, I do not want to audit the bar institutions, in the form of properties, I have written that: @inti @edicated public class Foo {@JoinTable (name = "T_FOO_BAR", joinColumns = @JoinColumn (name = "FOO_ID"), inverseJoinColumns = @JoinColumn (Name = "BAR_ID")) @ManyToMany (Cascade = Persist) @ Indited (targetAuditMode = RelationTargetAuditMode.NOT_AUDITED ) Public listing & lt; Bars & gt; GetBars () {Return bars; }} Now, I want to retrieve an amendment of foo : AuditReader reader = AuditReaderFactory.get (GetEntityManager ()); FU modification = (FU) reader.CreativeTeA () .entensEtrevision (Foo Class, 42) .getSingleResult (); Unfortunately, when I want to retrieve all the data (i.e. when it loads lazy times ), error me ORA-00942: table or view is not present , because Tried to query this: Sel...
Read more

mongodb - CakePHP paginator ignoring order, but only for certain values -

While pagging my user model in kppp, I can sort with some values, but not others. For example, I can order results from created or email , for example, but username or reputation seems to come back order by arbitrary command for example with a list of users: $ this-> paginate = array ('condition' => array ( 'User.is_active' = & gt; true), 'border' => 24, 'order' => array ('User.created' => DESC)); Works as expected, but $ this - & gt; Paged = array ('conditions' = & gt; array (' user.isIactive '= & gt; true),' border '= & gt; 24,' command '= & gt; array (' user repeats' = ' & Gt; DESC ')); No, no. I first thought that this could be a database issue, but when I search the database directly, sorted as expected. Note: I am using Mangodebi with the Mongo DB plugin of Ichikawa for the KPPHP; Pagnetizing users are used to work properl...
Read more