Scraping an Entire Website using LINUX

Web scraping (also called Web harvesting or Web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding certain full-fledged Web browsers, such as the Internet Explorer (IE) and the Mozilla Web browser. Web scraping is closely related to Web indexing, which indexes Web content using a bot and is a universal technique adopted by most search engines. In contrast, Web scraping focuses more on the transformation of unstructured Web content, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to Web automation, which simulates human Web browsing using computer software. Exemplary uses of Web scraping include online price comparison, weather data monitoring, website change detection, Web research, Web content mashup and Web data integration. [wikipedia]


Scarping a website using wget command in Linux.

$ wget -m --tries=7 "http://www.linuxnuggetz.blogspot.com"


Spoofing the browser that you are using.

$ wget --user-agent="Mozilla 2.0" -m http://www.linuxnuggetz.blogspot.com

Comments

  1. AnonymousOctober 23, 2010 at 7:07 PM

    its nice post i dont know how to extract web by using linux as basically i am from .net field but i got good info from your post thanks

    Web Data Extraction

    ReplyDelete
  2. RamseizeSeptember 8, 2011 at 9:32 AM

    nice tut, i would try this on my ubuntu box, if your still active in blogging, invite sana kita to join in our new blog community @ http://pinoygeeks.net

    ReplyDelete
  3. AnantzJune 5, 2012 at 5:01 PM

    Where does the scrapped website get stored?

    ReplyDelete
  4. UnknownJune 15, 2012 at 10:04 PM

    website will be stored in the current folder where you launch the wget command unless you specify another path with the "-O" (--output-document=FILE) parameter.

    ReplyDelete
  5. SJH_InvestJanuary 30, 2013 at 6:48 AM

    How do you install lynx and how well does it work with Ubuntu? I would like to gather property data for an auction coming up soon and this tool could help.

    Thanks

    ReplyDelete
  6. JagadeeshFebruary 24, 2014 at 1:18 PM

    how to scrap text alone from the website? .... as i am interested only in text processing i need to collect more text data.

    ReplyDelete

Post a Comment

Popular posts from this blog

Linux OS Backup and Recovery

Image
Source :http://relax-and-recover.org/documentation/getting-started This quick start guide will show you how to run Relax-and-Recover from the git checkout and create a bootable USB backup. Start by cloning the Relax-and-Recover sources from Github: git clone https://github.com/rear/rear.git Move into the rear/ directory: cd rear/ Prepare your USB media. Change /dev/sdb to the correct device in your situation. Relax-and-Recover will ‘own’ the device in this example. This will destroy all data on that device. sudo usr/sbin/rear format /dev/sdb Relax-and-recover asks you to confirm that you want to format the device: Yes The device has been labeled REAR-000 by the ‘format’ workflow. Now edit the ‘etc/rear/local.conf’ configuration file: cat > etc/rear/local.conf < Now you are ready to create a rescue image. We want verbose output. sudo usr/sbin/rear -v mkrescue The output I get is: Relax-and-Recover 1.13.0 / $Date$ Using log file: /home/jeroen/tmp/...
Read more

Radiator Radius Installation connecting to ORACLE (CentOS)

1. Download and install the Oracle client connectors. http://www.oracle.com/index.html oracle-instantclient-basic-10.2.0.3-1.i386.rpm oracle-instantclient-devel-10.2.0.3-1.i386.rpm oracle-instantclient-sqlplus-10.2.0.3-1.i386.rpm rpm -Uvh oracle-instantclient-basic-10.2.0.3-1.i386.rpm rpm -Uvh oracle-instantclient-devel-10.2.0.3-1.i386.rpm rpm -Uvh oracle-instantclient-sqlplus-10.2.0.3-1.i386.rpm 2. Set the environment variables. # ORACLE_HOME=/usr/lib/oracle/10.2.0.3/client/lib # export ORACLE_HOME # export LD_LIBRARY_PATH=$ORACLE_HOME # TNS_ADMIN=/usr/lib/oracle/10.2.0.3/client/admin # export TNS_ADMIN # mkdir /usr/lib/oracle/10.2.0.3/client/admin 3. Edit /etc/profiles and insert the lines below. ORACLE_HOME=/usr/lib/oracle/10.2.0.3/client/lib export ORACLE_HOME export LD_LIBRARY_PATH=$ORACLE_HOME TNS_ADMIN=/usr/lib/oracle/10.2.0.3/client/admin export TNS_ADMIN 4. Create a file "tnsnames.ora" and insert the settings below. SERVER1= (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = ...
Read more