Automatic use of web pages with .NET

Page 1 of 3

With the help of web browser macros, the need for more control over what happens during the automated use of a web page is becoming increasingly evident. After trying a number of possible solutions, C # and .NET provided the best option. Find out how.

There are many reasons why you might want to automate your interaction with a web page. In this project, the example, and this is a real example, is of an ISP providing a monthly bandwidth allowance. The problem is how much data has been used through the ISP’s website. Checking it manually is a time consuming chore. Automating work is made more difficult by the need to log in, navigate to the right page, and then extract the data. Although it can be done using the browser macro language, it does not provide an easy way to deal with the recovered data.

My first attempts were trying to build a JavaScript web application to do the job, but there are issues. JavaScript does not allow you to make a cross-domain Ajax call, so you cannot easily retrieve an arbitrary page on the web. There are ways around this problem, but they are all messy or require the cooperation of the owner of the website you are downloading.

After much thought, the best and fastest solution was to create a classic desktop application for Windows using C #. The complete list of the program is provided at the end of the article.

The key to the problem is the WebBrowser control / class. It is a wrapper managed around an ActiveX browser component. Back in the days when ActiveX was the way to create components, Microsoft implemented IE as an ActiveX module and it still is today. However, to use the WebBrowser, you don’t need to know how it works, you can just use it. What you need to know is that the WebBrowser control is based on the version of IE installed on the system and for some web pages it can make a difference.

Creating the web browser

Start a new Windows Forms C # project. To add a web browser to your project, you can simply select one from the toolbox and drop the control on the form. This has the advantage that you can size the control interactively and you can watch how the interaction goes because you can see the results of the page loading. However, in many cases after debugging the app you don’t want to see the web browser content at all. Since the web browser also uses a lot of resources, you might want to load and unload it as needed by the program.

Putting it all together suggests that creating the WebBrowser object in code is a good option.

First we need a global variable to reference the WebBrowser:

private WebBrowser webBrowser1;

and then we can create an instance whenever it is needed.

Add a multiline text box to the form and a button – the text box is used to display the results and the button starts the interaction with the web page.

Next, we need a makeBrowser method:

private void makeBrowser()
 webBrowser1 = new System.Windows.
 webBrowser1.Name = "webBrowser1";

While we don’t actually need a method to display the browser, it’s invaluable when debugging:

private void showBrowserInForm()
 webBrowser1.Location =
         new System.Drawing.Point(12, 300);
 webBrowser1.MinimumSize =
         new  System.Drawing.Size(20, 20);
 webBrowser1.Size =
         new System.Drawing.Size(783, 338);
 webBrowser1.TabIndex = 1;

This adds the control to the current form. We could have used a new form if you wanted the browser to display separately from the app.

Multiple pages

The first thing we need to understand is how to navigate to a page and respond when the page is loaded. Navigating to a page is just a matter of setting the URL property to a valid URL – it’s a bit more difficult to know when the page is fully loaded. When the document is loaded, it raises a DocumentCompleted event. The problem is that this event is triggered for every loaded frame. This means that you can get multiple DocumentCompleted events per page load. The solution to this problem is to test ReadyState and make sure it is set to Complete before processing the page.

void webBrowser1_DocumentCompleted(
   object sender,    
   WebBrowserDocumentCompletedEventArgs e)
 if ( webBrowser1.ReadyState!=
     WebBrowserReadyState.Complete) return;

However, this doesn’t always work, so be on your guard for attempts to process a page before it’s finished.

Next, we need to find a way to know where we are in the overall process of using the website. In most examples of using the WebBrowser control, only one page is processed. In this case, we have a login page, menu page, and results page to work with.

How do we keep track of the page we are dealing with?

One solution is to replace the event handler after each visit with an event handler that will process the next page. It’s a possible approach, but .NET doesn’t make it easy to change event handlers – you have to remove the old one, then add the new one.

An alternative approach is to write a single event handler that works for all pages and determines which one using the current URL. This has the downside that many websites change the URL you provide to something else or increase it with query parameters. URL checking may work, but it is not resistant to changes.

A simpler alternative is to use a “state machine” approach.

All we do is set a status flag on the next page to load and process. In general, the status indicator can be an object that holds several properties that determine the current state of the system, such as target URL, etc.

In this case, the status indicator can be just a simple variable:

private int page;

When the button is clicked, we first need to go to the site login page and set the status to 1, that is, we are dealing with the first page.

private void button1_Click(
             object sender,
             EventArgs e)
 page = 1;
 webBrowser1.DocumentCompleted += new
             "url for login page");

Note that it is up to you to provide the URL of the login page you want to work with. Also note that the DocumentComplete event handler is also configured.

Comments are closed.