Posts tagged Xml
Take Control of Messages in WCF
Dec 27th
With WCF you can control many aspects of how service behave and how you interact with them. Let’s take a look at Data & Message Contracts and how each behaves on the wire.
Data Contracts – The Basics
Data contracts are the default method for serializing in WCF. To leverage data contracts you simply markup the classes you wish to expose from the service with the DataContractAttribute. The data contract serializer does not support all the features of Xml Serialization which yields some performance boosts. However, the data contract serializer has some default behaviors which may not suit your needs.
The Contract
[DataContract] public class CompositeType { bool boolValue = true; string stringValue = "Hello "; [DataMember] public bool BoolValue { get { return boolValue; } set { boolValue = value; } } [DataMember] public string StringValue { get { return stringValue; } set { stringValue = value; } } }
The SOAP Message
Request (Showing soap:body only)
<s:Body> <GetDataUsingDataContract xmlns="http://tempuri.org/"> <composite xmlns:a="http://schemas.datacontract.org/2004/07/ContractExamples" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"> <a:BoolValue>true</a:BoolValue> <a:StringValue>Hello Service!</a:StringValue> </composite> </GetDataUsingDataContract> </s:Body>
Response
<s:Body> <GetDataUsingDataContractResponse xmlns="http://tempuri.org/"> <GetDataUsingDataContractResult xmlns:a="http://schemas.datacontract.org/2004/07/ContractExamples" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"> <a:BoolValue>true</a:BoolValue> <a:StringValue>Hello Service!Suffix</a:StringValue> </GetDataUsingDataContractResult> </GetDataUsingDataContractResponse> </s:Body>
With the data contract, notice how our messages are actually wrapped in an outer element in the Xml. For the request, the CompositeType is wrapped in a “GetDataUsingDataContract” element which is derived from the operation name. The response is where it is interesting, our message is actually double wrapped.
How do we change this default behavior? Enter Message Contracts.
Message Contracts – For Control Freaks
Message contracts are another way to work with WCF to take more control of how the messages are serialized in the SOAP envelope. These can be a key strategy to creating services that can support better interoperability with other stacks such as Java. You define message contracts with the MessageContractAttribute.
The Message Contract
[MessageContract(IsWrapped=true, WrapperNamespace=http://christopherDeweese.com/WCF/Examples)] public class CompositeTypeRequest { [MessageBodyMember] public CompositeType CompositeData { get; set; } } [MessageContract(IsWrapped=true, WrapperNamespace=http://christopherDeweese.com/WCF/Examples)] public class CompositeTypeResponse { [MessageBodyMember] public CompositeType CompositeData { get; set; } }
Here we have defined a request and a response message which contains our CompositeType. What result does this yield?
The SOAP Message
Request
<s:Body> <CompositeTypeRequest xmlns="http://christopherDeweese.com/WCF/Examples"> <CompositeData xmlns="http://tempuri.org/" xmlns:a="http://schemas.datacontract.org/2004/07/ContractExamples" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"> <a:BoolValue>true</a:BoolValue> <a:StringValue>Hello Service with messages! </a:StringValue> </CompositeData> </CompositeTypeRequest> </s:Body>
Response
<s:Body> <CompositeTypeResponse xmlns="http://christopherDeweese.com/WCF/Examples"> <CompositeData xmlns="http://tempuri.org/" xmlns:a="http://schemas.datacontract.org/2004/07/ContractExamples" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"> <a:BoolValue>true</a:BoolValue> <a:StringValue>Hello Service with messages! Suffix</a:StringValue> </CompositeData> </CompositeTypeResponse> </s:Body>
A Little ‘Philosophy’
In past posts on services I’ve discussed the opinion that you interact with services through messages. Encapsulating inputs and outputs to the service using messages allows for a clearer semantic representation of the what the capabilities the service provides access to.
Using Message contracts in WCF lets you explicitly control how the messages are serialized and allows you to build services from canonical data models created in raw Xml. This also provides the greatest interoperability and you don’t have to deal with the default behaviors of the data contract serializer.
Final Thought
You may not always need this type of control. For instance if your service is only being consumed by other .NET clients the messages may not be that important to you. But if you are designing services for an Enterprise SOA, then message contracts will be a great help.
NIEM Q&A – When to Use, Why, and Tips to Get Started
Dec 18th
Every month or so I receive questions about using NIEM in projects. The questions typically have the same theme, so I thought I would write up a quick post summarizing the questions I get most often and some answers to get you started.
Should I use NIEM in my project?
Do you need a single, canonical data model for exchanging data between systems? Do you have control of the systems you are integrating? Do you have an existing data model used within your systems? Is NIEM a requirement for your project/grant/contract?
If you answer ‘Yes’ to any of these questions then you may consider using NIEM. NIEM is best used where it will have the most benefit – describing exchanges of information between organizations & systems, harmonizing the language we use to talk to each other (i.e., Organization instead of Agency, Sur Name instead of Last Name), and defining requirements at the data exchange level. NIEM is not a one-size-fits-all tool and you will not get maximum ROI unless you have a strong use case for it and implement it properly.
Should I use NIEM for my database schema?
The short answer is probably not.
NIEM is an Xml data model and it’s best use is describing the data elements exchanged between systems. If you need to send data from System X to System Y then NIEM is a good candidate for doing this. You can define the elements you need to exchange and then build a NIEM IEPD to define the exchange. Using NIEM as a database schema does not always yield the best results. In most cases you are better off storing the data in an optimized way for your database platform and using adapters to translate data from the DB into NIEM.
How do I add new fields to NIEM?
In response to this question I would pose a different one – why do you need to add the fields? Can you derive them from other fields in the exchange? In most cases the 80% NIEM has built in will get you what you want. Fields like row-level timestamps or system specific ones may not be necessary and I would tactfully question those. If you find that you must add a field the best route to take is to define an extension schema in a custom namespace and add the fields there. Try to derive from base NIEM fields. For example, if you need to add a field to OrganizationType then derive your new type from OrganizationType and add the necessary elements.
Avoid adding the field and not specifying it in the schema properly. The pitfalls of doing so may not happen immediately but eventually one will and you’ll thank me for warning you.
I would question why you need to add fields as long as possible without delaying your implementation and deployment. In most cases you really don’t need the fields.
Can I use [Name_of_IEPD] for my project?
It depends. Does it have what you need? Are you ok reusing someone else’s work (one of the benefits of NIEM!), then yes. Sometimes existing IEPDs will be a good fit and get you the 80% or more. Other times the perspective may be vastly different from yours. You can certainly propose the use of an existing IEPD and have your business analysts and other technical staff review to see if it is a good fit.
The key to success with NIEM is that it brings together the business and technical teams involved in the exchange. If a particular IEPD is not able to bridge those gaps you may need to create your own. As NIEM adoption grows, more IEPDs are becoming available and many of them are coming from national groups that are looking at how things are done across the country to find the best 80% that fits most cases.
Getting Started with NIEM – Quick Tips
- Build a dictionary of the data you have or would like to have
- Search IEPDs to find one that matches your needs
- Map your data elements to the elements in the IEPD
- Store the data in a manner that makes sense for your database platform
- Translate your data from its source into NIEM and vice-versa
- Shoot for the 80% because in most cases that is what is really needed
More information about NIEM
For those of you working in the .NET Space check out my previous articles on NIEM + .Net
Visual Studio 2010 – Xml Schema Designer & Visualization
Nov 18th
While preparing for a presentation on Xml support in Visual Studio and Visual Basic I discovered the new Xml Schema Designer in VS 2010 Beta 2. The Xml Schema Designer lets you visualize your Xml Schema in a very interesting way that helps you see the structure and content defined by the schema set you are working with. Let’s explore it a little.
For this article I am using the Linq to NIEM sample I showcased at the NIEM NTE in October. I have converted the solution to VS 2010 Beta 2 and you can read more about that in this post.
After opening the LEXS 3.1 Schema, the following designer window is what you are greeted with:
This window allows you to bring items from your schema set into the schema designer. The designer reflected over the solution and located all the schema sets which you can see in the details list – 20 Schema documents, 820 Global Elements, 10 Global Attributes, 269 Complex Types, 57 Global Simple Types, 0 Global Model Groups, and 1 Global Attribute Group.
Clicking “add” on Global elements yields a scrollable view of all 820 Global Elements.
This is a little hard to work with and fortunately the VS team included some alternate views. My favorite is the content model view:
In content model view the elements you’ve added display on the left in the workspace. On the right is a scrollable region with all the elements and their children displayed individually. You can drill down further by selecting an element in the workspace and view only that element and it’s child nodes.
You can click on the
or
to expand or contract the child nodes allowing you to drill down all the way to the simple types.
If all that visualization wasn’t enough – how about we generate a sample? Right-click on an Xml node in the designer and make it so!
The sample contains data in every element and attribute, probably a bit of over-kill but gives you a nice view of what an instance document will look like.
Enjoy!
Visual Studio 2010 Project Conversion, Xml Intellisense Improvements, and a Few Other Tips
Oct 22nd
A few days ago the VS 2010 Beta 2 was announced and I snagged it from MSDN (MSDN is also sporting a new look and feel which is a nice change!). This evening I finally got around to installing it on my laptop. For Beta 1 I had installed it to my alternate boot-to-VHD copy of Windows 7 Ultimate. But I was feeling risky tonight and installed it on my main copy of Windows 7 Ultimate.
The full install weighed in at 6.4gb and included all the bits needed to start developing in Microsoft’s next evolution of Visual Studio.
To get started with VS 2010 I decided to take a copy of the Linq2NIEM project and convert it to 2010 to test out Xml Intellisense, MS Test Changes for VS unit testing as well as the VB Linq to Xml queries which I have grown very fond of in my previous work with NIEM Xml.
I hit a few snags during the conversion process:
- The copy of the project still included source binding files – those were removed by the conversion wizard.
- The Linq2Niem class library and Linq2NiemTests projects did not convert successfully with the solution. I had to manually convert each project.
- Post conversion there were target framework errors pointed out by the Test project. I changed the target framework on the Linq2Niem class library but discovered that you cannot change the target framework on a Unit Test project. This was a little disappointing, but I was able to just create a new project and import my existing test class file and it worked like a charm.
Other things I noticed:
- VS 2010 picked up my existing Vibrant Ink theme from my VS 2008 install. This was great because I would have hated to have to go manually pick all those colors
- Overall performance is improved from Beta 1, or so it seems. Intellisense seems more responsive and the Xml Intellisense seems much faster than in VS 2008.
- VB Linq to Xml Intellisense does not provide hints for Abstract element substitutions. The NIEM nc:DateRepresentation is a good example. nc:DateRepresentation can be substituted as nc:Date or nc:DateTime, but neither of those shows up as a choice. This is how things were in VS 2008 so it seems like a connect bug report/suggestion is called for here. You can work around it but you have to fight intellisense to get it to do what you need it to.
Figure 1: At this node the actual element should be nc:ActivityDate – but it’s not displayed
Figure 2: You can see in the query above – nc:DateTime is the appropriate element, however, only the abstract date representation is shown.
Figures 3 & 4 show how the Xml editor properly provides hints for the substitutable elements. Looks like the issue is really in the VB Linq to Xml parser.
Lastly, how about some gratuitous Windows 7 + VS 2008 “Vibrant Ink” porn? Looking good!
LINQ to NIEM On CodePlex
Oct 4th
Following up on the “IEPD Development in the .NET Environment” that I co-presented with Carl Nelson, I have posted the source code to the LINQ to NIEM example on CodePlex at http://linq2niem.codeplex.com/.
Look through the source and shoot me any questions you might have!
There is more NIEM and .NET goodness to come. Stay tuned!
Integrating Existing Systems With NIEM Using SOA – Follow Up
Sep 30th
As I mentioned in the beginning of the presentation, what I presented was based off my experiences and opinions and was subject to change and during the great Q&A and discussion, I received some great feedback and experiences from others that helped refine and redirect a few opinions I expressed.
Regarding adapters and whether they belong to the bus or to the systems being adapted for the bus – I see a case for both, but, I think that ultimately when you reach a point that you are adapting so many systems and doing so many transformations it makes more sense for those to be a concern for the bus. Maintaining that many adapters in individual systems would become a burden over time; something that only experience would teach you and it’s an experience I’ve not yet had and may not at this point.
Thank you to everyone who attended and stayed for the Q&A. I got a lot out of the Q&A and definitely have a lot more to learn. This was something different for me to present because I know where I think projects could’ve been done better if I had accepted the bus as a viable solution and if I had the lens of designing systems to be more event-driven. I wanted to share that perspective but also open up to other perspectives because that is where I think the sharing of experience really provides value to all. Thanks to the 4B Track Lead Anthony Hoang of DHS for the humbling introduction and coaching prior to the presentation.
Many props to guys like Dru Sellers, Udi Dahan, and Chris Patterson who have helped shape some of my thinking on the bus and events. Definitely check out their projects, Mass Transit and NServiceBus, as real solutions for those of you on the .NET side of the fence who need a functional bus to enable your EDA.
Here are the slides – I know the conference staff will post them, but I caved in and paid for Internet in the room so I’m going to use up all the bandwidth I can
Open Source NIEM Project on CodePlex
Jul 26th
I just published change set 25219 to the Logical Entity Exchange Specification (LEXS) .NET reference implementation. From the project page:
Project Description
A reference implementation for the Department of Justice Logical Entity Exchange Specification XML model which is based off the National Information Exchange Model. This project provides a reference for a LEXS implementation.
From: http://www.it.ojp.gov/default.aspx?area=implementationAssistance&page=1017&standard=486
The Logical Entity eXchange Specifications (LEXS) defines a common format in which information can be shared. The most commonly used elements form the foundation upon which practitioners can build specialized extensions to suit individual communities. LEXS 3.1 is based on NIEM 2.0.
The LEXS implementation is designed to demonstrate how to work with LEXS, a subset of the National Information Exchange Model using .NET. The initial efforts focus on the use of .NET classes and the XmlSerializer with code generated by WSCF.blue. The implementation includes:
- A full .Net class library of the LEXS types from the schema
- A functioning WCF service which meets the specs of the LEXS Wsdls and passes the soapUI test cases
- Interface types which define providers that can be implemented to support the LEXS Wsdl operations
There are no releases of this project yet but as soon as I finish up the "common" project I will post a release of the compiled assembly. Please feel free to peruse the source code for now and leave me some feedback / questions. The code is all in C# and uses some of the latest features like object initializers and lambda expressions (just for fun
Cheers!
WSCF.Blue Support For WCF and a Few Tips on WCF + XmlSerializer
Jul 12th
Over a week ago the Web Services Contract First (WSCF) project announced the "blue" version which supports WCF. WSCF is a great tool that saved my butt on a large integration project with one of our fine 3-letter agencies a few years back. The tool boasts the ability to take a WSDL and Schema files and turn them into C# or VB code. When I saw the new release I had to give it a try.
My first run went smooth but I noted the WSCF tool generated the service contract interface type without the "I" prefix. I reported this to the team and they promptly fixed it and even tried it against the DOJ LEXS WSDLs that I was working on. Many thanks to Christian Weyer and Alex Meyer-Gleaves for responding so quickly.
In the government data sharing space there is a lot of contract first development happening. The NIEM IEPD process encourages developing both the business need and data schemas before you begin coding the exchange. I think this is a great approach to take and helps everyone from the business analysts to the technical analysts understand the exchange, the data to be exchanged, and the overall process.
Having an open source tool like WSCF is a great addition to your toolbox when interop is key.
As I migrated a soon-to-be-released open source project to it’s new solution I used WSCF to generate all the classes and service stub. The following are some tips I have when you are using WCF + XmlSerializer; these really have nothing to do with WSCF, you will get the same if you use SvcUtil directly. That said I think WSCF generates the classes from the schemas with a lot less effort and actually better than Svcutil directly.
Tips for WCF + XmlSerializer
Tip 1: For types that are marked with a MessageContractAttribute and IsWrapped=True, if you inherit from a base class that exposes a public XmlSerializerNamespaces property the namespaces are not properly serialized. XmlSerializerNamespaces are used when you want to control the namespace prefixing and output during serialization. This is accomplished by creating a public field with the name "xmlns" and marking it up with the XmlNamespaceDeclarationAttribute. The XmlSerializer uses the presence of this field to output the Xml Namespaces and the associated prefixes within the element from the object in the object graph where the attribute was applied.
Tip 2: If you have a class marked with the MessageContractAttribute and you inherit from a base class within that class then the base class must also be marked with the MessageContract attribute. WCF gives you a nice error message explaining this because it is unable to start the service.
Tip 3: In some cases both SvcUtil and WSCF generate ReplyAction="*" on the WSDL. This could potentially make it fail in an interop scenario. Not exactly sure what causes that but in any case, if you have trouble then remove that from the service contract. A tell tale sign is that when you view the WSDL page for your service you do not see any of the operations listed.
Upcoming NIEM Training Events
Mar 22nd
Doris Girgis from the IJIS institute contacted me regarding some upcoming National Information Exchange Model (NIEM) events. If you are doing any government related IT work at the federal level or in many states chances are you will encounter projects using NIEM XML format to exchange data. This is especially true in the Justice field where NIEM is becoming the de facto standard for exchanging data when the data leaves your borders.
The following opportunities to learn more about NIEM are coming in 2009:
NIEM Practical Implementer’s Course – Ashburn, VA – April 14-16, 2009
Justice and Public Safety 2.0 – Ashburn, VA – June 2-4, 2009
NIEM National training event (pdf) – Baltimore, MD – September 30-October 2, 2009
There is currently a call for papers for the NIEM National Training event. You can see this document for more details.
Contact information for Doris or the other points of contact for each event can be found on in the links above.
LINQ to XML: Generating National Information Exchange Model XML Documents
Dec 18th
There are many ways to build XML documents but the main approach I have used in the past was to build an object model and leverage serialization. This approach was effective but costly in terms of serialization time and complexity of building the objects. I have previously blogged about using LINQ to XML to transform and parse NIEM XML documents but have not written about generating NIEM XML with LINQ.
For a new project at REJIS we were updating an existing interface to use NIEM XML. After some discussion with Sudhir Umarji I chose to use a pre-built N-Dex Booking IEPD. This provided the base with all the elements we would need to exchange. One of our senior developers began working with me to create a new service based solution.
Because we generally have a "vertical" market for our services we tend to build the service and the client. The approach we use is that the service returns the format that makes the most sense, which is always one of our domain object structures; a very simple object model for all the different documents and data elements we work with. The client uses a factory/adapter pattern where the adapters implement an interface and then through configuration we specify the concrete adapter to use at runtime. In code we create the instance using System.Activator. This gives us the most flexibility with outputting to different formats, databases, etc. One of our services using this pattern has three adapters in production and all are called from the same client program.
For this new service endeavor we would be using a similar approach but this time we would be using an adapter to transform records into the N-Dex Booking IEPD format. For this adapter I wanted to try generating the XML with LINQ. Since we work in VB.NET we have the advantage of using XML literals which turned out to be a pretty useful way for us to generate NIEM XML. It allows us to visually see the document we would be creating and break it into manageable chunks to control the output.
The problems encountered mostly revolved around managing the IDs and creating the associations as well as guarding against null elements in our object model and preventing them from outputting malformed XML. We also had to look a little into how to handle "for each" loops. This can be accomplished by using an embedded expression such as "<%= From person in MyRecord.People Select BuildPersonElement(person)%>" (Imagine here that BuildPersonElement takes a person object and outputs the appropriate XML).
Some tips:
- Identify the static parts of your document and break those into separate methods.
- A lot of structures in NIEM are very repeatable, identify those, and provide methods to assist in building them.
- Use good defensive programming in your methods to protect yourself from nulls. If your methods do not have all the required data elements to build the XML structure simply return nothing (unless it makes sense to throw an exception).
- Identify all your element IDs that will need to be created and manage those with lists.
- Use the VB.Net Import xmlns statements to import all your namespaces into your code file. This will help LINQ keep the namespaces prefixed and reduce the size of your XML.
- If you bring the schemas into one of the studio projects you can use that to get intellisense. For N-Dex based schemas this may not work because separate schemas that reference the same namespaces are used for the digest and structured content. We included the digest schemas and used studio to validate our XML instances.

