05/08/2019

How to Query Web Sites from Your Own Web Page

by Christos Karayiannis

To create a web page with an HTML form that queries a search-enabled web site, commonly a search engine or an e-commerce site, you have first to perform a query at the site in order to reverse engineer the underlying mechanism of its form.

Query for instance Google at your browser and test for a specific keyword, e.g. ‘network’.

When the search results appear you can find at the address bar of your browser that the Google’s URL:

https://www.google.com/

is replaced by a URL like the following:

https://www.google.com/search?q=network&oq=network&aqs=chrome..69i57j0l2j69i60j0l2.4419j0j7&sourceid=chrome&ie=UTF-8

From this URL you can make the following assumptions for the underlying data passing mechanism for the Google server:

Because the query changed the URL the HTTP request method used was GET.
By locating the ‘q=network’ part at the URL’s query string, which uses the name=value format, the name of the textbox where you have entered the keyword is derived to be ‘q’.
The program name that takes action at the server side when data are submitted is the one appearing on the left of the question mark just before the query string, that is ‘search’ and this is placed in the document root of the site (https://www.google.com/search).

The previous assumptions may also be verified by inspecting Google’s home page HTML source code and locating the form’s attributes.

Use now the information collected previously to create a web page and provide to the HTML tags of its form the appropriate values for querying Google. A minimal HTML source code that includes a textbox and a submit button would be creating with the following HTML source code:

^{<!DOCTYPE html>
<html>
<head>
<title>Searching on Google</title>
</head>
<body>
<form method="GET" action="https://www.google.com/search">
<input type="text" name="q">
<input type="submit" value="Google Search">
</form>
</body>
</html>}

At your desktop create a text file with an .html extension, e.g. called google.html and test the HTML form locally before publishing it. Open the HTML file with a text editor, enter the previous source code and save it. Next double click on file google.html to display the web page and use then its form to query Google. Web page google.html appears in the following figure:

New Content Item

By following similar steps you can also create HTML forms for querying Yahoo, Amazon, or any other site.

About the Author

Christos Karayiannis has taught Web Development courses for more than 20 years in High Schools and Institutes of Technology in Greece. He holds an MSc in Computer Science from the University of Wales and a Physics degree from Aristotle University. His main interests are Networking, Operating Systems, and Programming. Christos has contributed to Open Source projects by documenting source code.

This article was contributed by Christos Karayiannis, author of Web-Based Projects that Rock the Class.