The ‘visible web‘ is what one can find using general search engines such as Google, Ask, Yahoo etc. The ‘invisible web‘ is what internet users cannot retrieve using search engines. The invisible web is also called as deep web, deepnet, or the hidden web.

Way back in the year 2000, it was estimated that the invisible web or the deep web contained approximately 7,500 terabytes of data and 550 billion individual documents. Estimates based on extrapolations from a study done at University of California, Berkeley show that the deep web consists of about 91,000 terabytes. In contrast, the surface web or the visible web is only about 167 terabytes.

What comes under invisible web or deep web?

According to Wikipedia, the invisible web can be classified under one or more of the following categories.

• Dynamic content – dynamic pages which are returned in response to a submitted query or accessed only through a form, especially if open-domain input elements (such as text fields) are used; such fields are hard to navigate without domain knowledge.
• Unlinked content – pages which are not linked to by other pages, which may prevent Web crawling programs from accessing the content. This content is referred to as pages without backlinks (or inlinks).
• Private Web – sites that require registration and login (password-protected resources).
• Contextual Web – pages with content varying for different access contexts (e.g., ranges of client IP addresses or previous navigation sequence).
• Limited access content – sites that limit access to their pages in a technical way (e.g., using the Robots Exclusion Standard, CAPTCHAs or pragma:no-cache/cache-control:no-cache HTTP headers[citation needed]), prohibiting search engines from browsing them and creating cached copies.
• Scripted content – pages that are only accessible through links produced by JavaScript as well as content dynamically downloaded from Web servers via Flash or AJAX solutions.
• Non-HTML/text content – textual content encoded in multimedia (image or video) files or specific file formats not handled by search engines.

Read more about the invisible web on Wikipedia and on the UC Berkeley Library.

Image Credit: Flickr