Posts tagged NIEM
NIEM Q&A – When to Use, Why, and Tips to Get Started
Dec 18th
Every month or so I receive questions about using NIEM in projects. The questions typically have the same theme, so I thought I would write up a quick post summarizing the questions I get most often and some answers to get you started.
Should I use NIEM in my project?
Do you need a single, canonical data model for exchanging data between systems? Do you have control of the systems you are integrating? Do you have an existing data model used within your systems? Is NIEM a requirement for your project/grant/contract?
If you answer ‘Yes’ to any of these questions then you may consider using NIEM. NIEM is best used where it will have the most benefit – describing exchanges of information between organizations & systems, harmonizing the language we use to talk to each other (i.e., Organization instead of Agency, Sur Name instead of Last Name), and defining requirements at the data exchange level. NIEM is not a one-size-fits-all tool and you will not get maximum ROI unless you have a strong use case for it and implement it properly.
Should I use NIEM for my database schema?
The short answer is probably not.
NIEM is an Xml data model and it’s best use is describing the data elements exchanged between systems. If you need to send data from System X to System Y then NIEM is a good candidate for doing this. You can define the elements you need to exchange and then build a NIEM IEPD to define the exchange. Using NIEM as a database schema does not always yield the best results. In most cases you are better off storing the data in an optimized way for your database platform and using adapters to translate data from the DB into NIEM.
How do I add new fields to NIEM?
In response to this question I would pose a different one – why do you need to add the fields? Can you derive them from other fields in the exchange? In most cases the 80% NIEM has built in will get you what you want. Fields like row-level timestamps or system specific ones may not be necessary and I would tactfully question those. If you find that you must add a field the best route to take is to define an extension schema in a custom namespace and add the fields there. Try to derive from base NIEM fields. For example, if you need to add a field to OrganizationType then derive your new type from OrganizationType and add the necessary elements.
Avoid adding the field and not specifying it in the schema properly. The pitfalls of doing so may not happen immediately but eventually one will and you’ll thank me for warning you.
I would question why you need to add fields as long as possible without delaying your implementation and deployment. In most cases you really don’t need the fields.
Can I use [Name_of_IEPD] for my project?
It depends. Does it have what you need? Are you ok reusing someone else’s work (one of the benefits of NIEM!), then yes. Sometimes existing IEPDs will be a good fit and get you the 80% or more. Other times the perspective may be vastly different from yours. You can certainly propose the use of an existing IEPD and have your business analysts and other technical staff review to see if it is a good fit.
The key to success with NIEM is that it brings together the business and technical teams involved in the exchange. If a particular IEPD is not able to bridge those gaps you may need to create your own. As NIEM adoption grows, more IEPDs are becoming available and many of them are coming from national groups that are looking at how things are done across the country to find the best 80% that fits most cases.
Getting Started with NIEM – Quick Tips
- Build a dictionary of the data you have or would like to have
- Search IEPDs to find one that matches your needs
- Map your data elements to the elements in the IEPD
- Store the data in a manner that makes sense for your database platform
- Translate your data from its source into NIEM and vice-versa
- Shoot for the 80% because in most cases that is what is really needed
More information about NIEM
For those of you working in the .NET Space check out my previous articles on NIEM + .Net
Visual Studio/.NET/WCF/LINQ and NIEM Tips
Nov 10th
Over the years while working with .NET and NIEM I’ve run into a few kinks that require a setting tweak or an extra step to get it to work.
.NET General
Choices for working with NIEM Xml (From the .NET Base class library. There are third-party tools out there to assist as well):
- Serialization / Deserialization – building objects from Schemas using utilities like Xsd.exe, Wsdl.exe, SvcUtil.exe or an open source tool such as WSCF/WSCF.Blue.
- System.Xml – .NET Classes to work directly with Xml Structures. Includes X-Path, Xsd, etc.
- Linq to Xml – LINQ queries designed and optimized for Xml. VB.Net supports Xml Literals within class files and intellisense in Linq to Xml queries.
Linq
- System.Xml.Linq.XDocument, XElement, et. al. are not Serializable. You cannot accept them as an input or return them as an output from services (Web/Sockets/Queues/etc) or any place that will attempt serialize them.
- With Linq to Xml using XDocument or XElement changes the axis for your query. If you parse a string to an XDocument your query starts at the root node. If you parse a string into XElement your query starts from the first element in the Xml and not at the root. Typically this isn’t a major hassle but the problem is that you may not notice it until you actually try a query at run time and do not get the results you expect. (See also Unit Testing!)
WCF
- MaxReceivedMessageSize and MaxStringContentLength will be your enemies. Every service you write, you will want to change those values from the default 65536 otherwise callers to your service will get a nice Fault Exception when they send you a message larger than that.
- MaxItemsInObjectGraph will need to be adjusted if you are going to be serializing large objects using direct serialization or you try to return a very large list or array of objects.
- MaxArrayLength should be adjusted in conjunction with MaxItemsInObjectGraph. Typically you’ll need to adjust both. You’ll use arrays more if you do any interop work between .NET and other technology stacks as Arrays are typically safer for Xml serialization.
- For interop scenarios your best bets are to stay with simple data types supported by Xml Schema (Xsd): String, Date, Integer, and arrays thereof. Adding specific .NET types like GUID, List<T>, etc will add additional schemas to your service contract and may cause issues with other technology stacks.
- Data Contract Serialization (the default) gives you less control over your Xml messages. Xml Serialization gives you fine grain control and is usually a better fit for working with NIEM in WCF.
Visual Studio
- Visual Studio solutions only allow an Xml Namespace to be defined once. If you try to import two XSDs that define the same namespace you will not be able to leverage the Xml Intellisense and document validations provided by the Xml parser in Visual studio. This really comes into play if you attempt to import LEXS digest and payload schemas in the same solution.
- Typically I would create a project in a VS Solution just for Xml Schemas. You can use a class library and it is not necessary to reference it from other projects. Schemas are used at the solution level, even when the schemas are in a project.
- While navigating an Xml document if you type an opening bracket, select an Xml element, and press tab twice studio will create an Xml fragment with all the required elements.
These are certainly not all the tips, but its a start. Feel free to contact me if you have some to share or have any questions!
Tweets Preserved: NIEM NTE Tweetbook
Nov 7th
Yesterday afternoon @NIEMNTE posted: “#NIEM 2009 National Training Event Tweetbook is posted on-line! http://bit.ly/3kOE90”
This is the first I’ve seen of a Tweetbook and the concept is really a neat one. Weighing in at 84 pages the NIEM NTE Tweetbook collects the tweets from 52 individuals who were actively tweeting during the conference using the #niemnte hashtag. A great way to get a glimpse of what happened during those three days in Baltimore.
If you’ve never seen a Tweetbook or are curious as to what value tweeting could bring to a conference, give it a read. There are lots of great tweets highlighted that show how much you can say in 140 characters.
NIEM National Training Event 2009 – Focus on Architecture; Focus on Community
Oct 5th
Chris Deweese shaking hands with Kshemendra Paul, Federal Chief Architect (Yes it was Friday and yes that is his @meshirt)
Photo Courtesy Christina Bapst, DHS (using Anthony Hoang’s awesome camera)
The 2009 NIEM National Training event took place from September 30th to October 2nd in Baltimore, MD. I had the honor of attending this years event and delivering two presentations over the three days of the conference. You can view all of the sessions on-line with a free registration here. The sessions were great; a mix of hands on, practical NIEM implementation as well as informational sessions on NIEM and NIEM tools. Instead of typing up a review of the three days of sessions I will skip to the final half-day because that is where the prior days were summed up and the lessons learned and shared really came to life.
The final half-day of the 2009 NIEM National Training Event was a town hall meeting with the IT leaders of this country. Federal CIO Vivek Kundra, Federal Chief Architect Kshemndra Paul, DHS CIO Robert Spires, DOJ CIO Vance Hitch, and DHS EDMO Director Donna Roy. Vivek delivered the closing keynote and the remaining group fielded questions from the audience regarding NIEM, information sharing politics, the implementers community, and more in a town hall style meeting.
The questions were pointed and the answers were open and candid. The group is aware of where things have fell short and aware of where NIEM has enabled the ability to deliver business value at all levels of government. Throughout the closing keynote and the town hall, all members of the panel hit on several key success factors.
Speak to the Business in Terms of Business Value and Not Fancy Technical Terms
The business does not speak your language, they do not speak Xml, NIEM, SOA, IEPD. But they do speak value, and cost savings, vision, and innovation. Speak to the business in those terms. Don’t silo your self by speaking in your language.
“Architecture is a 12-letter word for planning.” –Ksmendra Paul
This was *the* quotable moment for an architect listening to this group. This quote was spoken but Mr. Paul himself, and the others on the panel echoed his sentiments. As federal CIO Vivek Kundra put it, architecture must be used to solve real problems or it becomes an academic exercise. SOA is designed to solve real problems and handle real complexity and that is at the center of the strategies the federal government is using to enable information sharing, reduce costs, and put NIEM to use as the data model standard for federal enterprise projects. These leaders talked about SOA, the bus, and how they are using it to provide value to the American people, reduce IT costs, and remain agile.
NIEM Needs a Community for Implementers to Keep the Grass Roots Effort Going
In a question that took me 10 minutes to build up the nerve to ask, I asked what the plans are to build a community for the implementers because it is the implementers that can help drive NIEM adoption in a way that speaks to the grass-roots which started it. Donna Roy fielded the question and candidly put it that the NIEM committee knows they have not reached out using the best means; using one-way communication with a static website and monthly news-letter. Donna said they need to leverage Web 2.0 and help us collaborate better and that they are working on it. And during her answer Donna paid homage to my humble beginnings as the NIEM Guy, when I started blogging about NIEM because no one else was and I had questions.
Community is where the real connections happen. When I was in front of a room that was 80% full of .NET developers looking to see how to connect the dots of NIEM and .NET, I realized how much of a community we need; because there are many out there who just need some quick answers to point them in the right direction of best practices and how to handle various situations. We need a community that will be open and will bring people in. As Donna phrased it, we need to be more like open source communities.
My pledge to everyone that is part of the NIEM community is that though I am leaving .gov employment, I will remain a part of the community and continue to contribute where I can to help others succeed in bridging the gap in information systems.
Along with all this is a shout out to all the #NIEMNTE Tweeps (in no particular order and I am sorry if I miss some of you!) – let’s keep the tweets going for #NIEM and #NIEMNTE.
- Datachick
- Malawto
- Akprakas
- nmccready
- g_christensen
- NIEMIfy (whoever you are)
- NIEMExecDir (Donna Roy herself!)
- webbr
- Bluemont (thanks for the lift to BWI!)
- Jim_Cabral
- jbiala
- orand
- gotNiem
- CarlCNelson
- NIEMBizArch
- Kurt_Cagle
- JoelByford
… and all the others I missed. Keep the community going!
LINQ to NIEM On CodePlex
Oct 4th
Following up on the “IEPD Development in the .NET Environment” that I co-presented with Carl Nelson, I have posted the source code to the LINQ to NIEM example on CodePlex at http://linq2niem.codeplex.com/.
Look through the source and shoot me any questions you might have!
There is more NIEM and .NET goodness to come. Stay tuned!
Integrating Existing Systems With NIEM Using SOA – Follow Up
Sep 30th
As I mentioned in the beginning of the presentation, what I presented was based off my experiences and opinions and was subject to change and during the great Q&A and discussion, I received some great feedback and experiences from others that helped refine and redirect a few opinions I expressed.
Regarding adapters and whether they belong to the bus or to the systems being adapted for the bus – I see a case for both, but, I think that ultimately when you reach a point that you are adapting so many systems and doing so many transformations it makes more sense for those to be a concern for the bus. Maintaining that many adapters in individual systems would become a burden over time; something that only experience would teach you and it’s an experience I’ve not yet had and may not at this point.
Thank you to everyone who attended and stayed for the Q&A. I got a lot out of the Q&A and definitely have a lot more to learn. This was something different for me to present because I know where I think projects could’ve been done better if I had accepted the bus as a viable solution and if I had the lens of designing systems to be more event-driven. I wanted to share that perspective but also open up to other perspectives because that is where I think the sharing of experience really provides value to all. Thanks to the 4B Track Lead Anthony Hoang of DHS for the humbling introduction and coaching prior to the presentation.
Many props to guys like Dru Sellers, Udi Dahan, and Chris Patterson who have helped shape some of my thinking on the bus and events. Definitely check out their projects, Mass Transit and NServiceBus, as real solutions for those of you on the .NET side of the fence who need a functional bus to enable your EDA.
Here are the slides – I know the conference staff will post them, but I caved in and paid for Internet in the room so I’m going to use up all the bandwidth I can 
Integrating Systems with NIEM using SOA
View more presentations from Chris Deweese.
Open Source NIEM Project on CodePlex
Jul 26th
I just published change set 25219 to the Logical Entity Exchange Specification (LEXS) .NET reference implementation. From the project page:
Project Description
A reference implementation for the Department of Justice Logical Entity Exchange Specification XML model which is based off the National Information Exchange Model. This project provides a reference for a LEXS implementation.
From: http://www.it.ojp.gov/default.aspx?area=implementationAssistance&page=1017&standard=486
The Logical Entity eXchange Specifications (LEXS) defines a common format in which information can be shared. The most commonly used elements form the foundation upon which practitioners can build specialized extensions to suit individual communities. LEXS 3.1 is based on NIEM 2.0.
The LEXS implementation is designed to demonstrate how to work with LEXS, a subset of the National Information Exchange Model using .NET. The initial efforts focus on the use of .NET classes and the XmlSerializer with code generated by WSCF.blue. The implementation includes:
- A full .Net class library of the LEXS types from the schema
- A functioning WCF service which meets the specs of the LEXS Wsdls and passes the soapUI test cases
- Interface types which define providers that can be implemented to support the LEXS Wsdl operations
There are no releases of this project yet but as soon as I finish up the "common" project I will post a release of the compiled assembly. Please feel free to peruse the source code for now and leave me some feedback / questions. The code is all in C# and uses some of the latest features like object initializers and lambda expressions (just for fun
Cheers!
Upcoming NIEM Training Events
Mar 22nd
Doris Girgis from the IJIS institute contacted me regarding some upcoming National Information Exchange Model (NIEM) events. If you are doing any government related IT work at the federal level or in many states chances are you will encounter projects using NIEM XML format to exchange data. This is especially true in the Justice field where NIEM is becoming the de facto standard for exchanging data when the data leaves your borders.
The following opportunities to learn more about NIEM are coming in 2009:
NIEM Practical Implementer’s Course – Ashburn, VA – April 14-16, 2009
Justice and Public Safety 2.0 – Ashburn, VA – June 2-4, 2009
NIEM National training event (pdf) – Baltimore, MD – September 30-October 2, 2009
There is currently a call for papers for the NIEM National Training event. You can see this document for more details.
Contact information for Doris or the other points of contact for each event can be found on in the links above.
LINQ to XML: Generating National Information Exchange Model XML Documents
Dec 18th
There are many ways to build XML documents but the main approach I have used in the past was to build an object model and leverage serialization. This approach was effective but costly in terms of serialization time and complexity of building the objects. I have previously blogged about using LINQ to XML to transform and parse NIEM XML documents but have not written about generating NIEM XML with LINQ.
For a new project at REJIS we were updating an existing interface to use NIEM XML. After some discussion with Sudhir Umarji I chose to use a pre-built N-Dex Booking IEPD. This provided the base with all the elements we would need to exchange. One of our senior developers began working with me to create a new service based solution.
Because we generally have a "vertical" market for our services we tend to build the service and the client. The approach we use is that the service returns the format that makes the most sense, which is always one of our domain object structures; a very simple object model for all the different documents and data elements we work with. The client uses a factory/adapter pattern where the adapters implement an interface and then through configuration we specify the concrete adapter to use at runtime. In code we create the instance using System.Activator. This gives us the most flexibility with outputting to different formats, databases, etc. One of our services using this pattern has three adapters in production and all are called from the same client program.
For this new service endeavor we would be using a similar approach but this time we would be using an adapter to transform records into the N-Dex Booking IEPD format. For this adapter I wanted to try generating the XML with LINQ. Since we work in VB.NET we have the advantage of using XML literals which turned out to be a pretty useful way for us to generate NIEM XML. It allows us to visually see the document we would be creating and break it into manageable chunks to control the output.
The problems encountered mostly revolved around managing the IDs and creating the associations as well as guarding against null elements in our object model and preventing them from outputting malformed XML. We also had to look a little into how to handle "for each" loops. This can be accomplished by using an embedded expression such as "<%= From person in MyRecord.People Select BuildPersonElement(person)%>" (Imagine here that BuildPersonElement takes a person object and outputs the appropriate XML).
Some tips:
- Identify the static parts of your document and break those into separate methods.
- A lot of structures in NIEM are very repeatable, identify those, and provide methods to assist in building them.
- Use good defensive programming in your methods to protect yourself from nulls. If your methods do not have all the required data elements to build the XML structure simply return nothing (unless it makes sense to throw an exception).
- Identify all your element IDs that will need to be created and manage those with lists.
- Use the VB.Net Import xmlns statements to import all your namespaces into your code file. This will help LINQ keep the namespaces prefixed and reduce the size of your XML.
- If you bring the schemas into one of the studio projects you can use that to get intellisense. For N-Dex based schemas this may not work because separate schemas that reference the same namespaces are used for the digest and structured content. We included the digest schemas and used studio to validate our XML instances.
