Data Exchange using XML and Delphi

Author: Deepak Shenoy

Introduction

• Introduction to XML
• Where does XML fit in?

XML and Delphi

•   Stock Demo - Steps 1 to 4
• Converting Existing Applications to use XML

Data communication using XML

• Interoperability with other systems
• Communication Demo

Technologies that use XML

• Internet Express
• BizTalk
• SOAP - Simple Object Access Protocol

Conclusion
References
Appendices

• Listing 1-Stock View Application
• Listing 2-Stock Server
• Listing 3-XML in a Client Dataset

-------------------------------------------------------------------------------------------------------

Introduction

You have a client-server application. It's the age old concept - a set of forms on the client, a database on the server. Data access through ADO or BDE. Works great on a desktop or on a corporate intranet.

Now, you have clients that want to access data "remotely" - across a WAN, or the Internet. So you're working on building a remote client for remote users. Then some specifications change, and you end up maintaining TWO versions - the remote app and the "local" application. Now someone wants a browser based client, and suddenly maintenance is a nightmare!

You obviously need a solution that reuses the code on your client, yet allows you to remotely access your server. There are many solutions that currently exist: I will not delve into all of them. I'll talk about how you can use Extensible Markup Language (XML) to exchange data between your server and client - a solution that I will demonstrate to be scalable to remote access, with reduced code-maintenance.

I will not demonstrate any proprietary code or even try to sell you any software - the aim of this article is to tell you how you can use XML to make your applications open and scalable. By open, I mean your application can "talk" to similar other applications, and by scalable I mean you can make remote clients, browser based clients etc. easier to develop.


I will compare my idea with Internet Express - the solution provided with Delphi 5. As it stands today, Internet Express does the job of getting your data on to a browser - a remote access solution. But there is one thing you lose - the ability to define your XML yourself, to import data from other XML - one of the main benefits of XML.

All of the source code accompanying this article is free for you to use, commercially or otherwise. Click here to download the source code.

XML– Introduction to XML

What is XML? It's a language. It is written is plain text - in a certain format. The "format" is used to structure the data so it makes it more readable than your average shopping list. Let's take an example: A list of books that you would want to display, for instance, could be shown in a paragraph listing the book name, the author and an abstract. In XML, you would describe such a listing as:

Figure 1

The parts between the "<" and the ">" is a "tag". Which means it isn't something that's part of the data itself, but something that describes what the data means. The piece of XML above reveals that there's a book named "Tomorrow won't be like yesterday", whose author is Pres EntTense and there's an abstract given. Now you can extract the fields you want and present it in your application. But before we go ahead, let's discuss how XML is structured.

XML Structure.

The prolog.
First, you need to be able to say "what follows is an XML document". Otherwise, a program wouldn't be able to tell it from anything else. You would add a prolog like so:

The only difference between the prolog and an XML tag is the "?" that's between the angle brackets.

The Document Element
Each XML document describes something - and this something is depicted as a base tag inside which all the other data goes in. In our example above, a book was the document element, so the XML was surrounded by <BOOK> and </BOOK>. If we had two or more books we'd have to use:

and <BOOKS> is the document element here. Every XML document needs one and exactly one document element.

Document Content
A document can various types of content

1. Elements
2. Attributes
3. Comments

Elements
A "tag" is an element. In our example above, BOOK, Author, Abstract are XML elements. Each element in XML should have starting and ending tags - for any tag X there must be a <X> and a </X> with the </X> coming after the <X>. In HTML, such a restriction is not imposed, so we sometimes find <p> tags which don't have a corresponding </p> tag.

Text between the element start and end tags is called the CDATA of the element. In,

<Soup> Today's special is the lobster soup </Soup>

The Soup element is described by the text between the tags, which is the CDATA. Sometimes, it's not necessary to have CDATA, like in <BR> tags in HTML. So one has to write

<BR></BR>

everytime, which is quite redundant. XML provides an abbreviated way to handle this: for elements with no CDATA, it's valid to start and end an element with a slash before the ending >, like <BR/>

Attributes

Attributes are properties of elements, which come inside the element tag, just after the element name. In,

<Product Name="Borland Delphi" Version="5"/>

Name and Version are attributes of the element Product. The attributes have values, as seen above. This kind of granularity can also be achieved by using sub-elements, since the above XML could be written as:

There's no real rule of thumb about which to use when. A general guideline would be to use attributes when the text is small and manageable, and sub-elements for larger texts or when the text may need to contain other sub-elements.

Comments
Comments in XML are just like HTML comments, starting with "<!--" and ending with "-->". Anything in the middle is ignored (even if there are any element tags in there, they will not be processed)

Document Type Definition (DTD)
XML, as we've seen, is just structured text. Since it is so, it needs to be parsed, and many parsers have been written for this purpose. Some ground rules need to be followed, such as:

a) always end an element tag. You can use only <TAG/>, <TAG></TAG> or <TAG>content</TAG>.

b) Maintain that there is only one document element surrounding the XML.

c) XML is case sensitive. So <Element> is not the same as <ELEMENT>

d) Use quotes when giving attribute values. Although

<img src=myimg.gif>

is valid in HTML, it is not in XML. It should be:

<img src="myimg.gif"/>

These ground rules form the basis for "well formed XML". But If you need to even define the structure of the XML, like which tag should follow which, what tags are allowed etc. you will need to describe that in a separate document, called a Document Type Definition (DTD).

Document Type Definition (DTD).

A DTD is also a text document. A DTD for our example above would be:

The first <!ELEMENT part defines an XML Element, in this case the <BOOK> XML element. The latter part of the same line (Author|Abstract) says that a BOOK tag contains either Author or Abstract tags. The <!ATTLIST part defines the Name attribute of the BOOK tag.The rest of the tags define the other elements in the XML.
An XML document can specify the DTD that defines it - like:

Note:(book-structure.dtd is a text file containing the dtd).

An XML that is well formed AND conforms to a DTD, it is said to be "valid XML". The XML that is used in this paper is not valid XML - DTDs have not been used anywhere, for clarity and space.

XML Schemas
An aside: DTDs were the first standard for defining XML. But a different parser has to be written for parsing a DTD, since a DTD does not conform to XML rules. XML schemas is a newer (although not yet a standard) approach, which uses valid XML to define another XML document. The format is different from a DTD, and supports some more data types. I won't delve into XML Schemas in this paper.

XML and databases
This demonstrates a structured set of data in plain text – so you can extract out the fields you want and present it in your application. Just like a table in a database, without the baggage of logging in, security, connection management and the like. And it's all plain text, so it's readable. XML is really extensible, unlike a table in a database - you can extend an XML to show master-detail relationships quite easily:

Figure 2.

This is a list of orders and the corresponding items. A file like this can be transmitted to a client over the Internet by email or using TCP/IP sockets. This can be used just like a database by the client, since the fields are neatly marked by tags. All of this, without the client having to own a license of the database server. Don't get me wrong: XML is not meant to replace database servers. It's going to complement them. You will still need optimized data storage, server side indexes, stored procedures and security - something XML will not address in the form it is now. (I wonder if this statement will remain sensible by the time this article is presented)

XML data can easily be passed over a network to a client who could choose to show it in a set of TDBGrids in a fat-client application. The same information can be formatted and shown in a Web Browser. (Internet Express does this job really well) You can even send XML to a customer-his payment system can now integrate with yours, all he needs to do is parse the XML, get a list of orders/items, confirm receipt, and generate payment information - all the customer needs to do is to sign the check. (If it were so easy...) And finally, the customer sends the payment information to you - again in XML - so you can directly import the details into your application for your information and later reconcile it with your bank statements.

Where does XML fit in?

Your client-server application consists of data being exchanged between your client and server. Data is exchanged in a format that you would not be aware of, which limits your application's scalability. If you used XML as the data exchange format, you could easily scale your client to a remote client – by simply implementing a data transport mechanism. You could use TCP/IP and have users accessing the server over the Internet, for instance.

In many enterprises, most user access consists solely of data reads. Instead of having all your users run client applications capable of reading and writing data, you can "web-enable"your server application by developing a web server application – an MTS component or an Enterprise Java Bean – that would connect to your server using TCP/IP or DCOM, get data in XML format and format it to emit HTML. This way all users can just log on to the Intranet site for reads. Later, you can even have other applications use this method to read data from the same server application and you will only need to understand the XML format returned. This makes it quite simple to build distributed applications.

Does this require that you give up developing applications using the BDE/ADO ? Or clients using data-aware controls? No. You will still use BDE/ADO on your desktop and Intranet Multi-tier applications.

I will show you how you can allow a client to access data on your server using XML. Which means that your server will need to have some components which can give information in XML and perhaps read information from a client in XML too.

How do you do this in Delphi?

Let's start looking at some code now. First, XML is plain text, so to make any real sense out of it,

XML needs to be parsed. Delphi 5 comes with a basic XML parser (in InternetExpress, but you cannot use it for other purposes), and there are many free and commercial XML parsers available.

I'll use the redistributable XML parser that comes with Microsoft Internet Explorer 5. It's not the best but it's definitely easy to start with since it supports COM interfaces. I've started off importing the Parser's interface from MSXML.DLL in the Windows System directory.

Figure 3.

I'll start with a demonstration here and extend it. First, I'll create an application that will show some stock quotes in a List View. (I've used some of the samples that Microsoft ships and converted them to Delphi. This is one of them, but I've extended it) The stock data is in an XML file.

Stock Demo

A small stock application that gets quotes from a server application - we'll use XML to communicate between the applications.

Step 1.
Let's now start with creating a small application that displays stock quotes to a user. The stock quotes are presumed to be in XML, such as:

Figure 4.

We'll drop a listview on the screen that shows each quote in a different row.A bitmap indicates whether the stock is up or down. The form looks like this:

Figure 5.

Listing 1 shows the code that creates the XML Parser, opens an XML file, parses it using the XML parser and displays it on screen. The XML file that was loaded was :

Figure 6.

The code that parses the document is :

Figure 7.

We've now got our first XML based client screen up and running

Step 2.
Let's move to the server. Obviously, we're going to have to lookup these stock quotes someplace. I'll make a dummy quote server now, which just looks up stock tickers from a database and generates random values for the stock prices.
I've created a table called Stocks with the structure in Figure 8.

Figure 8.

This stock quote server will have a port 3580 open for requests. A server socket will listen to requests, which have to be in XML of the form:

Figure 9.

The output will be XML (just as shown in Figure 6.), sent back to port 3580. This is pretty plain and simple. The server form is shown in Figure 1, and the code in Listing 2.

Figure 10.

The ADO Connection connects to an Access .MDB file that holds all the stock symbols
qrySymbol is an ADO Query that does "SELECT * FROM STOCKS WHERE SYMBOL=:SYMBOL"
The server socket listens for connections on port 3580. The code is given in Listing 2.

Step 3.
Let's now use a TClientSocket to send requests to this port at the server and receive the data.

Figure 11.

A screen shot of the update process is shown, with the refreshed data in view (Figure 12.)

Figure 12.

4. Showing the data on a Web browser : Here's code for an ISAPI DLL that will get requests from a web browser, and show the results to the user in a browser.

Figure 13.

A screen shot will give you an idea of what kind of output you can expect:

Figure 14.

Note: This is Internet Explorer 5, which can show XML.

This is a very simple example of what we can do with XML. If we have more stock servers that return quotes in the XML format in Figure 2, the client not depend on a particular server at all - just point the application to a different server if one is down for maintenance.

What if we wanted multiple users to edit data? Just place some code at the server that will parse the input XML and update the database - You might need to add validations, security etc. but the model is in place.

Here's the architecture of this application in a nutshell.

How do you convert currently existing applications?

You can't modify your existing applications to use this strategy, it'll take way too much time. Consider extending your current application to use XML. So you'll start off with exposing parts of your data using XML.

A large part of enterprise applications consists of only reads - there are much fewer instances of writing data to a database than to read from it. For plain reads, just run a query on the database and make XML out of it. You can do that quite easily by :

Figure 16.

This doesn't cover BLOB fields or Memo fields etc. But for float fields (use a DisplayFormat),
String/character fields and Integer fields this works quite well.

There is a possibility of an XML Dataset that could emerge soon. A TDataset descendant that will give you a way to show parts of XML in datafields. Then, you can start using XML to display your data easily. Of course, since XML is plain text, we still have the issue of speed with large datasets - but since you control the interaction between the server and the client, you can send data in chunks or even split it into more manageable parts.

Interoperability with other systems

With an XML based implementation, you have one more advantage. Your applications can communicate with other packages on potentially different platforms. If two airline companies were to collaborate – so their customers could have a single point of purchase. Lets say they used different software for invoicing. To avoid hauling one airline company on to a common invoicing system, you could design an application that would convert their data to a common XML format – and use this format in the common ticket order system. There are innumerable advantages to this approach – a saving in time and money is only one of them.

DEMO

I'll demonstrate two applications that use different data formats for internal storage. I've added XML support to both the applications so they can communicate with each other – they also have support for more such applications to integrate with them.
Note: All the source code is available with this document, suitably zipped.

First, take a look at the screen shot of an Internet Site that shows the list of Customers of Company ABC, which is a reseller of Product T, made by Company XYZ. This is XML, formatted for browser viewing using XSL - Extensible Stylesheet Language . (I will not cover XSL in this paper)

Figure 17.

Company XYZ direct sells Product T too, but it needs to see all the customers of the product. I've built an application that reads data from XYZ's internal database and shows it in a Grid. In addition, it reads the XML from the Company ABC's web site, and shows the customers in the same grid, shown in Red. (I've sorted the list by Company Name).

Figure 18.

Technologies and Products that use XML

InternetExpress
InternetExpress is an XML solution by Inprise. With the enterprise version of Delphi 5, (and C++ Builder 5, I guess) you can create web clients for your multi-tiered applications. InternetExpress comes with special Javascript libraries which can be used on any browser that supports Javascript. These libraries contain an XML parser too, but you may not use these for non-MIDAS applications.

You can create a regular Web application and drop a TXMLBroker on it. This acts like a ClientDataset, except it requests packets in XML instead. Then the XML is sent to the client, along with the Javascript XML Libraries. The XML is like this:

Figure 19.

The XML libraries at the client allow you to see the data, even modify (insert, update, delete) data without a round trip to the server. In effect, the XML libraries at the client cache these updates. You can then apply all the updates at one shot.

You can change the Web page layout - the HTML. But you have no control over the XML format. You cannot add custom tags, data or attributes directly.

Is it possible to connect to other systems or have other systems connect to your application?

Not unless they use MIDAS too. The XMLBroker component works with the IAppServer interface, an integral interface of MIDAS. Yes, you could have a customer read XML directly from your web server application using the format given above. (I'm still testing this theory) But XML isn't at its productive best, in my opinion.

If you are developing on MIDAS, try InternetExpress: it will save a lot of work if you want to have web clients.

BizTalk

In this context, Microsoft's Biztalk initiative (www.biztalk.org) is worth a mention. In Biztalk, you can define the XML definitions of your business objects. A worldwide definition library then allows to you to interact with other such publishers - now, with BizTalk server, you can have a PeopleSoft implementation connect to an SAP implementation etc. It's something worth looking at.

SOAP - Simple Object Access Protocol

There's a new kind of server coming up-The SOAP server. This have nothing to do with what comes on TV - it's much easier to understand, at the very least.

This concept came about because one had to distribute client access across machine and location boundaries. When a firewall steps in, it's usually more than hell to configure a DCOM based multi-tier application. One way out is to use the HTTP port - port 80. This is generally let through by firewalls. In fact, MIDAS uses this for distributed applications using a special ISAPI Dll that they have written. - httpsrvr.dll. The DLL transfers the calls to the registered objects on the server using DCOM. A big problem - the data format is proprietary. Which says, pretty bluntly, that you can only use MIDAS clients to access the server. What about the VB and VC++ developers out there that want to use the service? We'll have to get back to you on that.

SOAP is a protocol that defines a standard for passing data across to the server. The headers that come to the server are HTTP headers - but the body is XML. The XML is in a format defined by SOAP - but the only real limitation is that you need to enclose the XML in a certain set of tags and define your namespace. Currently, SOAP can be implemented using ISAPI Dlls on Windows NT, or through separate server programs.

If you were to communicate with a SOAP program, you would need to format your XML and send it to the server - this is something InternetExpress currently doesn't allow you to extend its functionality to. With the architecture I've outlined in this presentation, you can format your XML, send it to a SOAP server, parse and integrate the results.

By the time this article is presented, there may be commercial SOAP servers in place and there will be components to access them.

Conclusion

Adopt XML in your applications and you'll see the benefits in scalability. The technology has only recently become popular, but you'll find a lot more progress in this field – parsers, applications and servers that support XML. XML-Enable your applications as soon as you can – you need to start planning today.

References

1. Microsoft Windows DNA XML Resource Kit CD
2. Microsoft XML site (msdn.microsoft.com/xml)
3. Newsgroups:
- microsoft.public.xml @ msnews.microsoft.com
4. Mastering XML (Sybex) - Ann Navarro, Chuck White and Linda Burman, ISBN 81-7656-191-6
5. The XML Handbook (Addison Wesley) - Charles F. Goldfarb, Paul Prescod, ISBN 981-4035-87-4

Appendices

Here's the code to load an XML Document and display it in the List View:

Listing 2 - Stock Server

The server socket listens for a request, calls BuildXML and sends the returned string back to the client.

Listing 3 - Data inserted into a client dataset

-------------------------------------------------------------------------------------------------------