Accessing data from storage system using XAM API

If you are familiar with POSIX which is a collection of standards which enables portability of applications across OS platforms. It provide a standardized API for an OS, eg:- functions like 'open' to open a file, 'fdopen' to open stream for a file descriptor, fork to create a process etc. It defines file descriptor as a per-process, unique, nonnegative integer used to identify an open file for the purpose of file access. The Unix systems programming API is standardized by a large number of POSIX standards. Most operating system APIs share common functionalities like listing directories, renaming file etc.

XAM stands for "eXtensible Access Method". The XAM API will allow application developers to store content on a class of storage systems known as “fixed-content” storage systems.

The data centers have evolved into a complex ecosystem of SANs, fiber channels, iSCSI, caching technologies etc. Certain storage systems are efficiently designed to store the documents that donot change,  which is useful for preservation or archival. Digital imaging storage systems which us used to archive XRAY DICOM images, ie. documents that are not edited and to be kept for a long time need can use fixed storage. The applications accessing these solutions need to be written using only those vendor-specific interfaces. What if one changes their infrastructure? The application has to be rewritten. This scenario is an analogy to the need for JDBC/ODBC driver specifications as a standard API to interact with different vendors. XAM API specified by SNIA is the storage industry's proposed solution to the multi-vendor interoperability problem.

Consider EMC Centera, the data written to it is fixed in nature. Most common systems uses a file location (address) based approach to store and retrieve content. Centera has a flat address scheme called content address. When the BLOB object is stored, it is given "claim" check derived from its content which is a hash value (128-bit) known as Content Address (CA). The application needn't know the physical location of the data stored. The associated metadata like filename etc is added into an XML called C-Clip Descriptor File (CDF) with the file's content address. This makes the system capable of WORM. When one tries to modify/update a file, the new file will be kept separately, enabling versioning and unique access through its fingerprint checksum. Also, there is an essential attribute for retention that tracks the expiry of the file. The file can't be deleted until time surpasses the defined value.

This is a basic vendor specific architecture. They provides the SDK/API to acces the data. So the industry came up with standard in which XAM API provides the capability to access these systems independently.

XAM has concepts like SYSTEMS, XSETS, properties and streams ("blobs"), and XUIDs.

SYSTEM/XSYSTEM - is the repository

XSET - the content

XUID - identifier for the XSET

All these elements include metadata as key value pairs. There are participating and non-participating fields. Participating fields contribute to the naming of the XSET known as XUID. If XSET is retrieved from a SYSTEM and a participating field is modified, it will receive a different XUID. XUID created must be formed by using the content of the property or stream (e.g. run an MD5 hash over the value).

So a vendor specific library implements the XAM Application Programming Interface (API) which is known as XAMLibrary connecting to the Vendor Interface Module (VIM), which internally communicates to the storage system. Usually, VIMs are the native interfaces. Once the application load XAMLibrary and connect to the source, it needs to open the XSET by providing the XUID which will open up the content as a stream, called XStream. The XStream object is used to read the transported data which can be a query result or a file.

A sample code to read the data,

// connect

String xri = “snia-xam://"

XSystem sys = s_xam.connect(xri);

//open the XSet

XSet xset = sys.openXSet( xuid, XSet.MODE_READ_ONLY);

//getting metdata value for the key

String value = xset.getString( “com.example.key” );

XStream stream = xset.openStream( “com.example.data”,

XStream.MODE_READ_ONLY);

stream.close();

xset.close();

sys.close()

XAM fields have type information described using MIME types.
eg:-“application/vnd.snia.xam.xuid" for the an 80-element byte array, xam_xuid.

The XAM URI (XRI) is the XAM resource identifier specification associated to a vendor identifying the storage.

As XAM generates a globally unique name (address), the objects may move around freely in time, changing their physical or technological location so as to enable a transparent information lifecycle management (ILM). So the metadata information and retention can be externalized ie, even if the application is gone ie. retired, the information management can be handled through an API. This makes the data center more adaptive to organizational information management.

And, unlike the file systems, XAM tags the information with metadata and provides a technology independent namespace, allowing software to interpret the content independent of the application.

Architecture specification

http://www.snia.org/tech_activities/standards/curr_standards/xam/XAM_Arch_v1.01.pdf

Dissecting Protocol Buffers

There are many ways to serialize data to formats like XML, JSON etc. Google's ProtoBuf is a way of encoding structured data. It is the de facto standard at Google for any server-to-server calls.

The choice of using the method of object serialization depends on speed,size,compatibility, metadata integrity, platform independent etc which make Protocol Buffers a good candidate especially if it is used for Java (the code can also be generated for languages like C,Python etc).XML has been a standard as data interchange and serialization format for most of the applications. Java do have a native serialization mechanism for objects.PB is like IDL describing the entity or data structure. PB can be considered as a high-level language describing the input and output types. Then the compiler-generated code is used to hide the details of encoding/decoding from application code.Isn't this similar to CORBA or EJB? Hoi, its from Google. They have been using this binary encoded structured data as input,output,writable formats for map reduce.

Protocol buffers are advantageous because they support multiple languages (i.e., Python, Java, C++ and others), are cross-platform, flexible, and extensible. They are forward and backward compatible. It uses descriptive message (ie entity or object) definition files (.proto files). The proto files are parsed by the compiler provided (eg:protoc.exe) to generate the java files based on the message definition.This java code will have the "builders" for creating the object.Good thing is that the message definitions can be modified without affecting the parsers derived from an older legacy version of the .proto definition.If we consider XML for persistence, it will need a metadata associated with entity.But, the protocol buffers are self descriptive and deprived of such unecessary details making it smaller in size.

The entity will be defined as a message.The message can refer other messages, but need to be defined or imported.Several data types and repeated data are supported.
The available wire types are as follows:

WireType	Meaning	Used For
0	Varint	int32, int64, uint32, uint64, sint32, sint64, bool, enum
1	64-bit	fixed64, sfixed64, double
2	Length-delimited	string, bytes, embedded messages, packed repeated fields
3	32-bit	fixed32, sfixed32, float

It include the concept of optional elements: fields that aren’t currently needed are not included in the binary representation. This is similar to an XML shema for an element with minOccurs=0. XSD/DTDs provide data integrity. But I think portocol buffer can also provide data type definitions and values within itself without compromising integrity.

A key is associated with each data value.For the first 15 values/members in the structure this key is stored in 1 byte; 2 bytes is required for
each key representing the 16th through the 2047th member.

Take an example.Define a proto file,

message student{
optional int32 id = 1;
optional string name = 2;
}

Generate java file. Then save the data.

Student.student.Builder builder = Student.student.newBuilder();
builder.setId(100);
builder.setName("Hary");
builder.build().writeTo(output);

Then view the binary file.If you have an hex editor one can see the hex dump.Use vim, it has xxd.exe. Use it to generate the binary dump.

Byte 1: The key 00001000 give info like :
bits 2-5 ie. 0001 says the field number ie 1
bits 6-8 ie. 000 says the wiretype is 0 ie int32 (see the table)
Byte 2:
The value 01100100 is 100

Byte 3: The key 00010010
bits 2-5 ie. 0010 says the field number ie 2
bits 6-8 ie. 010 says the wiretype is 2 ie string (see the table)

Byte 4: The 00000100 says next 4 bytes for UT-8 character string

....

This is how the binary is file is structured. As protocol buffers include data binding library, it makes easy for encoding/decoding. It is faster (7x) than JSON serialization.Even unknown fields can be set to the object.

References

ProtoBuf HomePage
http://code.google.com/apis/protocolbuffers/docs/overview.html

NetBeans IDE Plugin for code generation
http://code.google.com/p/protobuf-netbeans-plugin/

Performance Using Internet data in Android applications
http://www.ibm.com/developerworks/opensource/library/x-dataAndroid/index.html?ca=dgr-twtrAndroidPArcedth-OS

Google Protocol Buffers - the Good, the Bad and the Ugly
http://www.freshblurbs.com/google-protocol-buffers-good-bad-and-ugly
MapReduce: A Flexible Data Processing Tool
http://cacm.acm.org/magazines/2010/1/55744-mapreduce-a-flexible-data-processing-tool/fulltext

Encoding
http://code.google.com/apis/protocolbuffers/docs/encoding.html

About HATEOAS

I think designing a REST URL is really challenging The resources are represented as hyperlinks.When REST applications are to be designed do business logic states depend on REST URL design ? The changes in application state are done through hyperlinks.So every possible state of applications are exposed as services and state changes are also exposed as urls.So we can draw a graph depicting change in states.So the transitions are done at run time while using the application.It is decided by the server.So the hypermedia as the engine of application state (HATEOAS) becomes a design constraint.So if the server responds as a set of links depicting the transitions, the consumer client can traverse through them to arrive at the final state... Concept seems pretty straightforward, but are there any underlying implementation complexities ? The relationship between resources can be logical, but the server makes relationships for the application.The client can have states defined by hypermedia elements.The tags like IMG,OBJECT,SCRIPT embeds resource within itself.The CSS can have image background links.So these urls can locate to the resource globally.The name space is defined by web itself.I think the scene becomes more interesting when web service discoveries,ordering of interactions, business logic making the client dynamic by the features of this concept.The libraries and frameworks as we know provide a way to call a single function and the system itself calls other functions internally.So this form of calling a single URL as a an entry point for calling other related urls linking logic or resource states performs same functionality like an interactive website.The urls can change at any time.The urls become a constraint in designing for a dynamic application.There will be a tight coupling for the resources and the client due to these permanent urls which will have to be residing forever.So the solution ? The response can be documents of urls for the client to traverse as i mentioned above as to call the resource from a single entry point... which is what HATEOAS is about

With a truly REST based architecture you are free to change just about anything about the disposition and naming of resources in the system, except for a few well known start point URIs. You can change the URI shapes (for example, if you decide that you really hate the currently scheme). You can relocate resources to different servers (for example, if you need to partition you data). Best of all, you can do those things without asking any ones permission, because clients will not even notice the difference.

http://barelyenough.org/blog/2007/05/hypermedia-as-the-engine-of-application-state/

More

http://sbtourist.blogspot.com/2009/01/jax-rs-and-hateoas.html
http://www.subbu.org/blog/2008/10/explaining-state-in-hateoas
http://www.syleum.com/2008/09/05/the-hypertext-constraint/
http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm#sec_5_3_3
http://www.infoq.com/articles/rest-introduction

Reading a cool book...