java - In Heritrix crawler tool how to extract the contents from crawled urls -

-

am new heritrix tool , able crawl web pages www , want extract contents of crawled urls.

please me one.please.thanks in advance.

 1.first download file  wget http://python.org/ftp/python/3.3.0/python-3.3.0.tgz or higher version root user.  2. change directory installed python  3. example /opt/python3.3/;  4. configure files ./configure --prefix=/opt/python3.3  5.make  6. sudo make install  7. /opt/python3.3/bin/python3  8.opt/python3.3/bin/pyvenv ~/py33  9.source ~/py33/bin/activate  10. wget http://python-distribute.org/distribute_setup.py  11.python distribute_setup.py    12. easy_install pip  13. pip install bottle  14. pip install warcat   15. if installed warcat check whether warcat installed or not.  16. python3 -m warcat --help after enter can see commands like, list,concat,extract etc..  17.python3 -m warcat list example/at.warc.gz  worked me ..enjoy

Comments

Post a Comment

Popular posts from this blog

Unable to remove the www from url on https using .htaccess -

-
i want remove www site url. ie. https://www.domainname.com.au https://domainname.com.au . i have tried various .htacces code nothing work me. here present code remove www http not work https. rewriteengine on rewritebase / rewritecond %{http_host} ^www\.(.*)$ [nc] rewriterule ^/(.*)$ http://%1/$1 [r=301,l] rewritecond %{https} off rewriterule ^(.*)$ https://domainname.com.au/$1 [r=301,l] rewriterule ^code-of-conducts/(.*)/$ code-of-conducts.php?type=$1 rewriterule ^parents/post-a-job/(.*)/edit/$ parents/post-a-job.php?action=edit&id=$1 rewriterule ^parents/post-a-job/(.*)/$ parents/post-a-job.php?job_type=$1 rewriterule ^parents/open-jobs/(.*)/delete/$ parents/open-jobs.php?action=delete&id=$1 rewriterule ^parent-profile/(.*)/$ parent-profile.php?id=$1 rewriterule ^parents/job-details/(.*)$ parents/job-details.php?applied_id=$1 rewriterule ^sitterprofile/(.*)/$ sitterprofile.php?id=$1 rewriterule ^sitter/job-board/state/(.*)/page/(.*)/$ sitter/job-board.ph...
Read more