Don't Directly Serialize Your Classes To JSON or XML

Web services, AJAX backends, file vending and receipt, messaging, and many other use cases call for the creation and consumption of structured data, i.e. data with rich shape and not necessarily a flat collection of scalar values. There are many tools and techniques (like JAXB) in practically every popular language to somehow go from a JSON or XML document to a hydrated object. Yet every day, new tools and techniques (for example, those using Java annotations to drive externalization behavior) arrive on the scene and we constantly seem to spend more time than we wish in this area of development.

The challenges aren't in the tools. "Edge conversion" of content continues to be a time-drain for these reasons:

1. There is only one, single compile-time version of code but there could be many externalized forms.
   It's not the simple, strict-rules conversions that cause the trouble; it's when the rules have to be broken to accommodate special object transformation and/or encodings, consumer demands, representations that combine other objects and potentially dynamic value generation logic, and other variations. Simply put, you cannot closely couple the object with forms of externalization. If you do, the first conversion pathway (from object to a particular output shape in a particular format) might be easy to produce, but as you add more, you quickly run into these problems:
   • Assuming decoration through annotations (or similar features in languages other than Java), the externalization logic gets complex as version of externalization format now has to be carried in the annotation, and it is unclear if technically more than one of such annotations can exist in one runtime environment.
   • Different tools available for "quick and easy" externalization (e.g. Jackson's POJO) have different target type and definition/declaration semantics. Proper design of get/set methods, null field handling, parent/child containering, and static initialization issues may be mutually exclusive across tools.
   • The unbounded set of tools to externalize the class will drag in a plethora of dependencies. Assuming for the moment that none of these clash with either other externalization tools or core dependencies of the object itself, they still introduce a heavy dependency profile to what could otherwise be a simple class.
   • The expanded compile-time dependency and complexity significantly increases the test & deploy cycle for no change in the basic, internal function of the class.

2. Objects and most languages have strong definitions of types, names, and values but externalized forms have different interpretations of these or sometimes lack them altogether.
   Objects have methods and state and there is no guarantee of one-to-one correspondence between a method and an underlying private piece of data.

   JSON has names and values but a limited type system; in particular, since it has no record definitions (schema), this means that to properly round-trip data to JSON back to data again, something has to know somewhere that a collection should be bound to a particular language-specific type.

   XML is simply a pile of nested tags and no intrinsic definition of type, name, or value. All semantics for the consumption of the tags as structure are implied and realized by a particular implementation of parsing/processing software. The XML specification says nothing about name and value, only the rules on what characters can be used in tags. In particular, because XML does not natively understand the concept of a collection, the distinction between name and type gets very distorted and in kind, so does conversion to- and from-object representation. Consider this simple XML data:

       <data>
           <name>Buzz</name>
       </data>
   

   Now, let's add a hobby:

       <data>
           <name>Buzz</name>
           <hobby>hacking</hobby>
       </data>
   

   How about ANOTHER hobby? We have two choices. The first is unfriendly XML:

       <data>
           <name>Buzz</name>
           <hobby>hacking</hobby>
           <hobby>cars</hobby>
       </data>
   

   It is unfriendly because until the second hobby is set up, a consumer does not know if hobby is modeled to be an entity occuring 0 or 1 times.

   The second choice is friendly XML where it is made structurally clear that hobby can occur more than once:

       <data>
           <name>Buzz</name>
   	<hobbies>
             <hobby>hacking</hobby>
   	  <!-- does not matter if "cars" is present or not.
   	       We "know" that there can be more than 1 hobby
   	       -->
   	</hobbies>
       </data>
   

   This is clearer, yes, but now look what happens:
   • What is the relationship between hobbies and hobby other than the fact that we used the plural form to "wrap" the instances of hobby?
     Fact is, there is none. We could have wrapped the hobby collection with grimblepritz; it is just a piece of text. How do we really know that hobbies is a collection of hobby?
   • When converting this XML into an object, which field is driving the show, i.e. is this a collection of hobby or a list hobbies?
   What ends up happening is the contained element (in this case hobby) takes on a blended, implicit role of both name and type. It is a name because it is used as a key to find it's value but it is also a type in the context of wrapper that implies it lives in a collection.

3. JSON and XML have very different handling of collections, and a single environment for creating/consuming these two formats from a common set of objects is difficult.

4. XML complicates the issue by supporting attributes which are not in the same namespace as nested elements. This means the following XML is perfectly legal:

       <data type="mineral">
           <type>vegetable</type>
       </data>
   

   and there is no default way to represent both type datums in a single JSON structure, although one could do somthing like this:

       {
           "@type":  "mineral",
           "type":  "vegetable"
       }
   

5. JSON has no "enclosingTag" but in XML it is required. For example:.

       {
           "msg": "hello"
       }
   

   The JSON above will parse into (typically) a Map, e.g.:

       Map m = parseJSON(stream);
       String s = (String) m.get("msg"); // s = "hello"
   

   But in XML, an equivalent structure would be:

       <data>
           <msg>hello</msg>
       </data>
   

   Parsing this XML introduces an extra level into hierarchy.

The Solution

The Bespoke Objects

    package com.whatever.bespoke;

    public class MyClass {
        private List<Transitions> tlist;
        private Actor actor;
        private Map<String,Money> paymentThreshold;

	// ... various high-value methods
    }

The Map

    package com.whatever.maputils;

    import com.whatever.bespoke.MyClass;
    import java.util.Map;
    import java.util.HashMap;
    import java.util.List;
    import java.util.ArrayList;

    public class MyClass2Shape1 {
        public static Map toMap(MyClass c) {
            Map m = new HashMap();
            List l = new ArrayList();
            for(Transition t : c.getTransitions()) {
              l.add(TransitionMapUtils.toMap(t));
            }
            m.put("transitions", l);

            Map m2 = new HashMap();
	    for(Map.Entry me : c.getPaymentThreshold()) {
                m2.put(me.getKey(), MoneyMapUtils.toMap(me.getValue()));
	    }
            m.put("thresholds", m2);
            // ...
        }
    }

1. MyClass2Shape1 is in the maputils package, NOT in the bespoke package
2. MyClass2Shape1 contains only static methods to do conversions
3. Some of the other bespoke objects like Transition have ther own mapping utilities (TransitionMapUtils, MoneyMapUtils) that can be used to piece-wise contruct an outbound Map shape
4. The Map may only contain those types that can be reasonably converted to a target externalizable format. String, Date, Integer, Long, and BigDecimal figure prominently here (plus Map and List). It largely becomes a Map of scalars for which toString() is very well defined.
5. The to/fromMap functionality depends only on the bespoke objects and basic Map and List functionality which is clearly available in Java and pretty much any other language. No third party dependencies or special code of any kind is required here; it's only about understanding what a pre-stringified representation of the object might look like.
6. Development of a mapper is admittedly tedious but straightforward. The good news is incremental additions to the mapper are very easy because there is likely already an existing snippet that can be copied and modified.

The Converters

    package com.whatever.jsonutils;

    import whatever.json.implementation.is.required;

    public class JSONUtils {
        public static void writeJSON(Map m, OutputStream os) {
            // whatever implementation is required
        }
        public static Map parseJSON(InputStream is) {
            // ...   
        }
    }

and

    package com.whatever.xmlutils;

    import whatever.xml.implementation.is.required;

    public class XMLUtils {
        public static void writeXML(Map m, OutputStream os) {
            // whatever implementation is required
        }
        public static Map parseXML(InputStream is) {
            // ...   
        }
    }

1. The converters live in package namespaces discrete from the the Map layer. This is important to prevent any third party physical dependency from creeping into the otherwise technology-neutral middle tier.
2. As with the Map utils, the output converters are static functions that take in a stream and produce a Map or take a Map and write it to a stream.

The Consequences

1. At a high level, the dependency/variability spectrum is polarized so that the bespoke objects and converter levels are relatively UNchanging, and the Map layer is as extendible as necessary. There is "no limit" to the number of MyClass2ShapeN variations that can be crafted. Remember, it is not unreasonable to have more than version of XML or JSON in production to satisfy consumers -- but there is only one version of the production software!

   	Layer
   	Bespoke	Map	Converters
   Depends On	Nothing except other bespoke objects	Bespoke objects Basic scalar types, Map, and List	Basic scalar types, Map, and List Third-party libs for parsing and/or emission, if necessary
   Changes	Only when methods critical to the business function of the object are added/modified	As often as necessary to create new shapes of externalizable data	Rarely after the first set of converters formatters is created
   Variations	None	As many as necessary to create new shapes of externalizable data	Few

2. Adding a new converter is immediately leveragable to converting many existing Map shapes. Making the Map the externalized data form keeps the complexity to O(N) where N is the number of converters instead of O(N²) (recoding the formatter for each bespoke type).

3. Objects organize their internal data on a single axis; that is, there is a single namespace of data within the object (sidestepping composition of other classes for the moment). Specifically, objects do not have a data section separate from an attributes section. So, when the MyClass2ShapeN mapper is run, it yields a very straightforward Map, also with a single axis. And since the XML converter is merely walking the Map, you end up with very friendly XML. There are no attributes and all data is structured in exactly the same way. Some might balk at the idea of XML with no attributes but if you're serious about consistent and straightforward deserializtion into a Map, it quickly becomes clear that attributes only get in your way.

4. Clearly, lists of data do not exist in software in a form similar to the unfriendly XML form described earlier. You cannot have two items named hobby in the same key-value collection.
   A reasonable strategy for list emission in the XML converter is to take the name of the list, append _LIST to it, and use the result as the wrapper tag. For example, given this:

       Map m1 = new HashMap(); 
       List

   we will end up with a main map m1 that has a name things which references a List. Note that the elements of the List are Maps and are essentially untyped and unnamed; they are just integer offsets. Just for grins, here's what the JSON generator would emit:

       {
           "things": [
             { "name": "value1" },
             { "name": "value2" }
            ]
       }
   

   JSON exactly follows the in-memory Map represention. XML isn't so lucky, but using the strategy described above, an XML converter would emit:

       <someEnclosingTag>
         <things_LIST>
   	<things>
   	  <name>value1</name>
   	</things>
   	<things>
   	  <name>value2</name>
   	</things>
         </things_LIST>
       </someEnclosingTag>
   

   The same converter module could also be sensitive to _LIST when asked to parse XML and upon detecting it, would "promote" the resulting List into the parent map and then remove the _LIST element. So instead of this:

       Map m1 = result.get("someEnclosingTag");
       Map m2 = m1.get("things_LIST");  // m2 is a "wasted step"
       List l2 = m2.get("things");
   

   we would have:

       Map m1 = result.get("someEnclosingTag");
       // things_LIST has been removed 
       List l2 = m1.get("things");  // ah!  things is now in parent (m1); no wasted step
   

   Of course, we probably wouldn't hard code _LIST in the handler but instead would code a "list strategy" class and pass that in, something akin to this:

       public class MyXMLListStrategy implements XMLListStrategy {
           public void produceList(Context ctx) {
               // Upon production of list, candidate wrapperName
               // is the name of the item in the Map, in the example
               // above, "things" 
               ctx.wrapperName = ctx.wrapperName + "_LIST";
               ctx.containedName = ctx.wrapperName;
           }
           public void consumeList(Context ctx) {
               int zn = ctx.wrapperName.indexOf("_LIST");
               if(zn > 0) {
                   // Shorten the wrapper name:
                   ctx.wrapperName =  ctx.wrapperName.substring(0, zn);
                   ctx.collectNestedElements = true;
               } // else not one of our constructs; consume it unchanged
           }
       }
   
       // In use:
       XMLUtils.writeXML(out, map, "nameOfEnclosingTag", new MyXMLListStrategy());
   

   In the example above, the list strategy is "Map-centric" in that the emitted XML is named with a bias toward the Map structure. Some consumers might want something more "XML-centric" where the wrapper tag has a more familiar plural or similar feel to it and the contained items are named so as to represent a single item:

   
       // This turns:
       //     people: [ "A", "B" ] 
       // into the familiar:
       // <people>
       //   <person>A</person>
       //   <person>B</person>
       // </people>
       public class MyXMLListStrategy implements XMLListStrategy {
           public void produceList(Context ctx) {
               if(ctx.wrapperName.equals("people")) {
                   // wrapper remains people; instead we change the contained name
                   ctx.containedName = "person";
               }
           }
           public void consumeList(Context ctx) {
               if(ctx.wrapperName.equals("people")) {
                   // wrapper remains people; instruct the engine to collect
                   ctx.collectNestedElements = true;
               }
           }
       }
   

   In general, the XML list strategy logic for emission takes a list name and returns two things: a) the name to use for the list    b) the name to use for each of the contained items. The logic for list parsing takes a list name and returns the new name to use for the list into which all the contained items will be collected.

5. It is acceptable for the map created from a bespoke object to have different (and probably richer) types than a map hydrated from XML or JSON or other format. For example, a mapper might take a date in a bespoke object and place it in a Map as a Date, but after conversion to JSON and round-tripping to a new Map, the date will be carried as a String (mainly because JSON does not have a Date type). The reason this is not a showstopper is that the mapper class knows that the target field in the bespoke class is a Date, so if not presented with a Date, it can do whatever it needs to try to construct a Date from the input material. In essentially all cases, some form of a String can be used to round-trip a richer type (Date, BigDecimal, etc.) for a particular conversion.

   Note that both JSON and XML have schema facilities that live outside the core definition of the formats. JSON has json-schema (see json-schema.org and XML has many mature options including XSD. More sophisticated mappers and converters can make use of these facilities as desired, and when used, proper and consistent types can be generated on "both sides" of the process.

Like this? Dislike this? Let me know