Posts tagged LINQ
Loving Linq & Generics: Using .Except with EqualityComparer<T>
Jan 12th
In the last post on Loving Linq we covered the .Except method for comparing two lists. The example was very simple and demonstrated how to compare to String lists. In this post we’ll cover an advanced scenario where we need to compare lists of a class we created.
We’ll start with an implementation of IEqualityComparer<T> EqualityComparer<T> which consists of overriding two methods: Equals and GetHashCode. What was a little tricky about this implementation is this subtle, but critical, comment in the code sample on MSDN (in the Enumerable.Except documentation):
// If Equals() returns true for a pair of objects,
// GetHashCode must return the same value for these objects.
public class WidgetComparer : EqualityComparer<Widget> { public override bool Equals(Widget x, Widget y) { if (ReferenceEquals(x, y)) return true; return x.Id == y.Id && x.Name == y.Name && x.Price == y.Price; } public override int GetHashCode(Widget obj) { if (ReferenceEquals(obj, null)) return 0; var nameHash = obj.Name == null ? 0 : obj.Name.GetHashCode(); var idHash = obj.Id.GetHashCode(); var priceHash = obj.Price.GetHashCode(); return nameHash ^ idHash ^ priceHash; } }
Now that we have our EqualityComparer set up for the Widget class, let’s look at how the Except method uses our implementation.
Test 1 – Two lists with same reference items.
[TestMethod] public void Except_ListsWithSameReferenceItems_ShouldNotYieldDifferences() { var widget1 = new Widget { Id = 1, Name = "Widget1", Price = 0.99 }; var widget2 = new Widget { Id = 2, Name = "Widget2", Price = 0.99 }; var list1 = new List<Widget> { widget1, widget2 }; var list2 = new List<Widget> { widget1, widget2 }; var results = list1.Except(list2, new WidgetComparer()); Assert.IsTrue(results.Count() == 0); }
Note in our implementation (which follows the example from MSDN) we check object.ReferenceEquals in the Equals method. But that check alone will not suffice, the result from GetHashCode is also validated so we must calculate the HashCode uniformly for the object as well.
Test 2 – Two lists with same items (but not same references)
[TestMethod] public void Except_ListsWithSameItems_ShouldNotYieldDifferences() { var list1 = new List<Widget> { new Widget { Id = 1, Name = "Widget1", Price = 0.99 }, new Widget { Id = 2, Name = "Widget2", Price = 0.99 } }; var list2 = new List<Widget> { new Widget { Id = 1, Name = "Widget1", Price = 0.99 }, new Widget { Id = 2, Name = "Widget2", Price = 0.99 } }; var results = list1.Except(list2, new WidgetComparer()); Assert.IsTrue(results.Count() == 0); }
In this test we use widgets set up with identical properties for each list. The test will still pass because our EqualityComparer evaluates the widgets using the formula we specified (Id, Name, & Price all match).
Test 3 – Two lists with different items. This one is tricky though, because it List 2 is the one that contains different items.
[TestMethod] public void Except_List2WithDifferentItems_ShouldNotYieldDifferences() { var list1 = new List<Widget> { new Widget { Id = 1, Name = "Widget1", Price = 0.99 }, new Widget { Id = 2, Name = "Widget2", Price = 0.99 } }; var list2 = new List<Widget>{ new Widget {Id = 1, Name = "Widget1", Price = 0.99}, new Widget {Id = 2, Name = "Widget2", Price = 0.99}, new Widget { Id = 3, Name = "Widget3", Price = 0.99 } }; var results = list1.Except(list2, new WidgetComparer()); Assert.IsTrue(results.Count() == 0); }
What gives? When you call Except you are saying “Give me everything that is in list 1, that is not in list 2.” In this case, list 2 has more items than list 1 and all the items from list 1 are in list 2.
Test 4 – Two lists with different items. This time list 1 has more items than list 2.
[TestMethod] public void Except_List1WithDifferentItems_ShouldYieldDifferences() { var list1 = new List<Widget> { new Widget { Id = 1, Name = "Widget1", Price = 0.99 }, new Widget { Id = 2, Name = "Widget2", Price = 0.99 }, new Widget { Id = 3, Name = "Widget3", Price = 0.99 } }; var list2 = new List<Widget>{ new Widget {Id = 1, Name = "Widget1", Price = 0.99}, new Widget {Id = 2, Name = "Widget2", Price = 0.99} }; var results = list1.Except(list2, new WidgetComparer()); Assert.IsTrue(results.Count() == 1); }
As expected, we find that Widget3 does not exist in list 2. Remember, when you call Except, say the mantra: “Give me everything that is in list 1, that is not in list 2.”
If you have any comments or improvements on this simple example of using Except, post away!
Loving Linq: Use .Except to Find the Difference Between Two Lists
Jan 3rd
While working on a side project for a friend who needed a program to compare two files, I discovered a handy extension method called ‘Except’. This method allows you to take two lists and find the set difference between the two. For strings, this was pretty straight-forward and I just followed the example from MSDN.
Example
var differenceQuery = sourceList.Except(externalList);
The result of this query will be an IEnumerable of the type from each list (in this case it was a string). The list will contain everything from “sourceList” that does not exist in “externalList”.
What do you do if you need to check lists of your own classes?
Just implement an IEqualityComparer<T> for your class. We’ll cover that in a future post!
Visual Studio/.NET/WCF/LINQ and NIEM Tips
Nov 10th
Over the years while working with .NET and NIEM I’ve run into a few kinks that require a setting tweak or an extra step to get it to work.
.NET General
Choices for working with NIEM Xml (From the .NET Base class library. There are third-party tools out there to assist as well):
- Serialization / Deserialization – building objects from Schemas using utilities like Xsd.exe, Wsdl.exe, SvcUtil.exe or an open source tool such as WSCF/WSCF.Blue.
- System.Xml – .NET Classes to work directly with Xml Structures. Includes X-Path, Xsd, etc.
- Linq to Xml – LINQ queries designed and optimized for Xml. VB.Net supports Xml Literals within class files and intellisense in Linq to Xml queries.
Linq
- System.Xml.Linq.XDocument, XElement, et. al. are not Serializable. You cannot accept them as an input or return them as an output from services (Web/Sockets/Queues/etc) or any place that will attempt serialize them.
- With Linq to Xml using XDocument or XElement changes the axis for your query. If you parse a string to an XDocument your query starts at the root node. If you parse a string into XElement your query starts from the first element in the Xml and not at the root. Typically this isn’t a major hassle but the problem is that you may not notice it until you actually try a query at run time and do not get the results you expect. (See also Unit Testing!)
WCF
- MaxReceivedMessageSize and MaxStringContentLength will be your enemies. Every service you write, you will want to change those values from the default 65536 otherwise callers to your service will get a nice Fault Exception when they send you a message larger than that.
- MaxItemsInObjectGraph will need to be adjusted if you are going to be serializing large objects using direct serialization or you try to return a very large list or array of objects.
- MaxArrayLength should be adjusted in conjunction with MaxItemsInObjectGraph. Typically you’ll need to adjust both. You’ll use arrays more if you do any interop work between .NET and other technology stacks as Arrays are typically safer for Xml serialization.
- For interop scenarios your best bets are to stay with simple data types supported by Xml Schema (Xsd): String, Date, Integer, and arrays thereof. Adding specific .NET types like GUID, List<T>, etc will add additional schemas to your service contract and may cause issues with other technology stacks.
- Data Contract Serialization (the default) gives you less control over your Xml messages. Xml Serialization gives you fine grain control and is usually a better fit for working with NIEM in WCF.
Visual Studio
- Visual Studio solutions only allow an Xml Namespace to be defined once. If you try to import two XSDs that define the same namespace you will not be able to leverage the Xml Intellisense and document validations provided by the Xml parser in Visual studio. This really comes into play if you attempt to import LEXS digest and payload schemas in the same solution.
- Typically I would create a project in a VS Solution just for Xml Schemas. You can use a class library and it is not necessary to reference it from other projects. Schemas are used at the solution level, even when the schemas are in a project.
- While navigating an Xml document if you type an opening bracket, select an Xml element, and press tab twice studio will create an Xml fragment with all the required elements.
These are certainly not all the tips, but its a start. Feel free to contact me if you have some to share or have any questions!
LINQ to NIEM On CodePlex
Oct 4th
Following up on the “IEPD Development in the .NET Environment” that I co-presented with Carl Nelson, I have posted the source code to the LINQ to NIEM example on CodePlex at http://linq2niem.codeplex.com/.
Look through the source and shoot me any questions you might have!
There is more NIEM and .NET goodness to come. Stay tuned!
LINQ to XML: Generating National Information Exchange Model XML Documents
Dec 18th
There are many ways to build XML documents but the main approach I have used in the past was to build an object model and leverage serialization. This approach was effective but costly in terms of serialization time and complexity of building the objects. I have previously blogged about using LINQ to XML to transform and parse NIEM XML documents but have not written about generating NIEM XML with LINQ.
For a new project at REJIS we were updating an existing interface to use NIEM XML. After some discussion with Sudhir Umarji I chose to use a pre-built N-Dex Booking IEPD. This provided the base with all the elements we would need to exchange. One of our senior developers began working with me to create a new service based solution.
Because we generally have a "vertical" market for our services we tend to build the service and the client. The approach we use is that the service returns the format that makes the most sense, which is always one of our domain object structures; a very simple object model for all the different documents and data elements we work with. The client uses a factory/adapter pattern where the adapters implement an interface and then through configuration we specify the concrete adapter to use at runtime. In code we create the instance using System.Activator. This gives us the most flexibility with outputting to different formats, databases, etc. One of our services using this pattern has three adapters in production and all are called from the same client program.
For this new service endeavor we would be using a similar approach but this time we would be using an adapter to transform records into the N-Dex Booking IEPD format. For this adapter I wanted to try generating the XML with LINQ. Since we work in VB.NET we have the advantage of using XML literals which turned out to be a pretty useful way for us to generate NIEM XML. It allows us to visually see the document we would be creating and break it into manageable chunks to control the output.
The problems encountered mostly revolved around managing the IDs and creating the associations as well as guarding against null elements in our object model and preventing them from outputting malformed XML. We also had to look a little into how to handle "for each" loops. This can be accomplished by using an embedded expression such as "<%= From person in MyRecord.People Select BuildPersonElement(person)%>" (Imagine here that BuildPersonElement takes a person object and outputs the appropriate XML).
Some tips:
- Identify the static parts of your document and break those into separate methods.
- A lot of structures in NIEM are very repeatable, identify those, and provide methods to assist in building them.
- Use good defensive programming in your methods to protect yourself from nulls. If your methods do not have all the required data elements to build the XML structure simply return nothing (unless it makes sense to throw an exception).
- Identify all your element IDs that will need to be created and manage those with lists.
- Use the VB.Net Import xmlns statements to import all your namespaces into your code file. This will help LINQ keep the namespaces prefixed and reduce the size of your XML.
- If you bring the schemas into one of the studio projects you can use that to get intellisense. For N-Dex based schemas this may not work because separate schemas that reference the same namespaces are used for the digest and structured content. We included the digest schemas and used studio to validate our XML instances.

