Web scraping (also called Web harvesting or Web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding certain full-fledged Web browsers, such as the Internet Explorer (IE) and the Mozilla Web browser. Web scraping is closely related to Web indexing, which indexes Web content using a bot and is a universal technique adopted by most search engines. In contrast, Web scraping focuses more on the transformation of unstructured Web content, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to Web automation, which simulates human Web browsing using computer software. Exemplary uses of Web scraping include online price comparison, weather data monitoring, website change detection, Web research, Web content mashup and Web data integration. [wikipedia]
Scarping a website using wget command in Linux.
$ wget -m --tries=7 "http://www.linuxnuggetz.blogspot.com"
Spoofing the browser that you are using.
$ wget --user-agent="Mozilla 2.0" -m http://www.linuxnuggetz.blogspot.com
Scarping a website using wget command in Linux.
$ wget -m --tries=7 "http://www.linuxnuggetz.blogspot.com"
Spoofing the browser that you are using.
$ wget --user-agent="Mozilla 2.0" -m http://www.linuxnuggetz.blogspot.com
its nice post i dont know how to extract web by using linux as basically i am from .net field but i got good info from your post thanks
ReplyDeleteWeb Data Extraction
nice tut, i would try this on my ubuntu box, if your still active in blogging, invite sana kita to join in our new blog community @ http://pinoygeeks.net
ReplyDelete