4.1 Introduction

This chapter demonstrates how to pull data from the Web and use it within your own applications. As mentioned in Chapter 1, Web pages are hosted on computers that run Web server software such as Microsoft Internet Information Services (IIS) or Apache. Hypertext transfer protocol (HTTP) is used to communicate with these applications and retrieve Web sites.

There are many reasons why an application may interact with a Web site, such as the following:

• To check for updates and to download patches and upgrades

• To retrieve information on data that changes from hour to hour (e.g., shared values, currency conversion rates, weather)

• To automatically query data from services operated by third parties (e.g., zip code lookup, phone directories, language translation services)

• To build a search engine

• To cache Web pages for faster access or to act as a proxy

The first half of this chapter describes how to send and receive data to web servers. This includes an example of how to manipulate the HTML data received from the web server. The chapter is concluded with an implementation of a custom web server, which could be used instead of IIS.