Pages

Showing posts with label open source. Show all posts
Showing posts with label open source. Show all posts

JiBX - Part 1

When we consider XML as a data model while programming, focus is mostly given on strings/tags/elements in parsing data.The ETL operations are done on this ubiquitous document interchange model when communication is made between applications.To make things simple in the internet; which is fond of documents ; XML became the foundation of services through web.The object, which is accessed from object graphs in the memory have features such as inheritance / polymorphism and attributes / object-based relationships while XML does not have any of these features as objects have.It is merely a grammatical (hierarchical) representation of data with all its branches mangled to itself .But they have similarity in the sense of representation of real world data.Therefore they can exists as the representation of each other which eases in programming.They are effective in defining business uses cases.But there is an impedance mismatch between objects and xmls.An application written in Java, will have it data types defined within the scope while XML Schema which defines the data is richer than Java.Complex objects find difficulty in serializing.Have a look on this paper which explains the X/O impedance mismatch.

We can generate Java classes from XML or vice versa.In this case, it is a "Schema-Centric" approach.We define an XML schema.Then one or more XML documents.Then generate java classes based on them.In this case you need a stable schema which can be used to validate data.Essential for a "reliable" web service.But application code is forced to use an interface that reflects the XML structure which makes them tightly coupled to contracts defined.Any change in schema make a need to regenerate the object model and change application code to match.

If we map the classes using bindings, then its "Java technology-centric " approach.This can be adopted when you don't want object model tied to schema.When there is a need to support different versions of the schema with the same object model or a common data exchange format for existing objects is needed.Binding definitions are themselves XML documents, with a schema defined for it.

JiBX is fundamentally a Java technology-centric (mapped binding) approach uses binding definitions.You define how XML relates to Java objects.Start from schema, code, or both.Binding code is compiled (or wired ?) into files (which uses BCEL byte code enhancements ). It can be done at build time, or on-the-fly at runtime.This makes JiBX compact and fast.JiBX achieves its performance by using post-compilation bytecode manipulation rather than the reflection.The advantage is that there is no need for getters, setters and no-arg constructors.One can write the class without considering mapping issues and then map it without modifications. XML schema describe domain objects.JiBX really maps well to XSD. JiBX is based on a XML pull parser architecture (XPP3) . Rather than generating code from a DTD or Schema, JiBX works with a binding definition that associates user-supplied classes with XML structure. The binding compiler is executed after code is compiled and the marshalling/unmarshalling code is added on to the class files.There are tools along with it, like Bindgen that can generate schema from existing classes.


Tutorials

http://jibx.sourceforge.net/binding/tutorial/binding-start.html
Pdfs - JiBX -Part1 JiBX-Part2 Intro

Previous articles
Creating a Java WebService using Axis 2 (a lazy approach)
A simple RSS parser

A snippet on Apache Sling

We know everything is content.Content(Knowledge) management systems are widely used in various fields and is really an old concept.Usually data is stored in databases or file system as it is very easy to manage unless data is huge..When we consider filesystem, it is unstructured and hierarchical.Database is structured with a well defined schema but restricted by constraints and transactional behavior.Web 2.0 caused such an insurge of highly dynamic data into web.It became inevitable for collaboration and integration.If we have to access information of different
formats, then we can use content repositories; for its inherent nature of file system and database (with some extra features).Its a best of both worlds.Here is the definition of a content repository from from JSR-170 spec

A content repository is a hierarchical(tree like) content store with support for structured and unstructured content, full text search, versioning, transactions, observation, and more(they can be considered as content services).

It is like a high level information management system.It can have both text files as well as binary files like pdfs,docs,images etc

Apache Sling (with a new release of version 5) is an open source Web framework for the Java platform designed to create content-centric applications on top of a JSR-170-compliant (aka JCR) content repository like Apache Jackrabbit.If you use Sling, it will be easy to develop large websites with thousands of pages.It works on top of JCR.If we are familiar with configuration management systems like changeman,VSS or subversion we know how it is easy to manage content.It avoids spreading of files in webserver Sling process the HTTP request in a RESTful way.Sling uses scripts or servlets to render and process content.Modularization by OSGi (using Apache Felix ) concept which loads modules to memory dynamically.Sling uses JSP,ESP (server side ECMA script),Groovy etc.We can directly map URL to a content.I think I have to know more about OSGi to dig whats happening inside Sling.I believe Sling could be used in large websites like social networks.DevDay says how Open Social and Apache Sling fit together.

more on Sling later...

Apache Lucene - Indexing Part2

I was going through some interesting sections of Apache Lucene these days.I found it really interesting because, the project is a very popular one and it made most of the web applications to integrate complex search modules in them.Some might be knowing JIRA tracker by Atlassian uses lucene which traverses huge buglists,comments,codes,documents etc.
For a vanilla search tool, comparing the search key with strings in the file is very slow.So the indexing like inverted index comes in handy.When indexing is done by lucene, it will create document ids for each document.It will collect all the words and associate them with each docId in which the word belongs. Therefore, each docId will be having alist of positions of words in the document.The index datastructure, the store of documents, with its associated fields is constructed to provide a random access data retrieval.The Lucene inverted index can be either opened to add more documents or delete existing documents at a time.
To update a document you must delete it first, close the index and add it again.
The Analyzer , specfied in the Indexwriter, will extracts the tokens to be indexed.There is a default analyzer for english texts(for multilingual one custom analyzers are needed).Before analyzing is done, the documents like pdf,doc etc are to be parsed.A Term is the basic unit for searching. Similar to the Field object, it consists of a pair of string elements: the name of the field and the value of that field.A term is defined as a pair of <fieldname,text>A term vector is a collection of terms.The inverted index map terms to documents.For each term T , it should store the set of all documents containing that term.So the duty of analyzer is to look for the terms in documents and create a token stream so that they can be mapped.Terms are stored in segments and they are sorted.The term frequency will tell how well that term describes the document contents.But term which appear in many documents are not very useful for filtering.The Kth most frequent term has frequency approx 1/K ie for 100 tokens, the index will contain 50% text.For the indexing strategies : - they can be chosen from
  • Batch based - like a simple file parsing and sorting-
  • BTree - indexing - similar to indexing by file systems and databases - as it is a tree the update can be done in place
  • Segment based which is common, created by lots of small indexes
The algorithm used for lucene indexing can be
  • indexing a single document and merging a set of indexes
  • incremental algorithm in which there will be a stack of segments and new indexes are pushed to stack (segment based)

Apache Lucene - Indexing - Part 1

"Information retrieval (IR) is the science of searching for documents, for information within documents and for metadata about documents, as well as that of searching relational databases and the World Wide Web."

Most of the application uses search features.If you are looking to add a powerful text search engine feature to your application then use Lucene, which can add advanced Search Engine capabilities to an application.This is a really powerful Java API which gave birth to powerful tools such as Nutch,Hadoop,Hibernate search and so on.Lucene was started in 1997 and adopted by Apache in 2001.The main functionality Lucene does is the powerful full text indexing of data.
Indexing with Lucene breaks down into three main operations: converting data to text, analyzing it, and saving it to the index.Lucene looks for strings only , so the documents has to be parsed and indexed.
To search large amounts of text quickly, you must first index that text and convert it into a format that will let you search it rapidly, eliminating the slow sequential scanning process. This conversion process is called indexing, and its output is called an index. So the searching is done on this index to find the data related with a cost of space 'storing indexes'.
These index files can be stored in a directory .A lucene index is divided into segments madeup of several index files(Lucene Documents).An index can be related to mutiple documents.So if new documents are indexed , it is added to segments than modifying the existing index file.Lucene uses a feature called incremental indexing ie there will be a global indexing and index those incremental documents so that it is searchable.Regarding the structure of a lucene index, it is an inverted index .While searching, lucene loads the index to memory .It uses a high performance indexing which has an index size roughly 20-30% of the size of text indexed which uses less memory. The documents in an index is a collection of fields which is a named collection of terms like <field,term>.These fields are independent search space defined at run-time.The segments or sub-indexes are independently searchable and the results of these segments are merged.Suppose a wiki article is indexed , we can set the field properties, so that the field objects contain actual indexed article data or stored one.



More about lucene index file formats - here

An account of Open Source Summit 2008 Hyderabad

During this weekend I attended the Open source Summit held at IIIT Hyderabad which was on 13-14 (I was unable to attend the event on the second day :( ). On Saturday morning I took the MMTS from the station nearby home and came to HafeezPet and from there reached IIIT by auto around 11 am.


The first session I attended was on BeleniX, opensolaris LiveCD project - Moinak Ghosh,I arrived the conference room, while the presentation was halfway around . The presenter was upgrading the openSolaris and while it was going on, other applications were executed !! He was explaining how openSolaris and ZFS is useful in production ready environment. He demonstrated creating separate snapshots. He explained about using DTrace, which can be used to dynamically inject debug codes while the application is running (can be used for debugging kernel). He explained about the difference between zones in Open Solaris and virtualization, concept of RAMDisk etc. The session was good as practical samples are demonstrated.


Next session was more interesting was, by Mahesh Patil from National Ubiquitous Computing Research CenterCDAC, Embedded Linux and Real time Operating systems. I really enjoyed and understood the technology. When I was in my college (MCET Trivandrum) we used to conduct lot of seminars. Sensor networks, nano technology were most presented those days. But this session as a great experience as he had to show something cool. He had a board with ARM processor and he demonstrated loading the Linux OS into it. He explained about ToolChains , how it can be used, packaging the kernel images etc. He described how an embedded OS is different from RTOS and about the preemptive nature of RTOS.RTOS uses the dual kernel approach in which the interrupt handling latency can be reduced by a kernel handling it and other operations by other kernel. The core kernel operations are given a low priority as the other tasks which are to be executed with higher priority in the queue. I came to know that most of the embedded Linux is based on POSIX compliance, but in Japan it is MicroItron. He explained about ECOS a configurable OS which can be configured for embedded or real time. Then about the Smart Dust project, cool futuristic technology; tiny devices floating around which communicate within a small range where they sleep most of the time. I was wondering about how huge the data will be produced by these devices. Think about real-time heat maps of different boxes holding vaccines those are distributed around the world! (Now pharmaceutical companies have a device kept inside the package to record the temperature when it was packed and check the change in temperature when opened) .Also came to know about the 3KB Linux – TinyOS ! Cool and simple… even though I am not from electronics background ...


On to the stage, was a geek – Arun Raghavan from Nvidia.He is a developer in Gentoo Linux community.I hasn't tried this Linux variant before. It's a linux for developers!! Any application can be customized for performance, maintainability creating ebuilds which will make it so flexible for developers. I think it will have a good learning curve as most of the installations and customizations can be done by a user .He demonstrated creating ebuilds for gtwitter a twitter client.He demonstrated the ease of using Portage which is a package management system used by Gentoo Linux.Visit Gentoo.org to know more about this linux.I really liked the article written by Daniel Robbins(Architect of Gentoo Linux) about the birth of it; read here .


I attended another session which was on Hadoop by Venkatesh from Yahoo research team.Hadoop is an opensource project for large data centers .I was looking forward for this presentation as it is about the web 2.0 (cloud computing) and large scale computing (blogged before). It is a framework written in Java that supports data intensive distributed applications.To know more about large scale data processing using Hadoop you can read this paper.It has a file system called HDFS filesystem ( pure-Java file system !!) stores replicated data as chunks across the unRaided SATA disks.There is a name node and a cluster of data nodes like a master-slave system.Name node stores an inverted index of data stored across the file system.Concept is similar to Google File System and its cluster features.More about the concept here (Nutch) , here This framework can be used for processing high volume data integrated with lucene will help to create a quality search engine of our own.This framework is used by Facebook. In one of the Engineering @ Facebook's Notes explains why Hadoop was integrated . Read here. It is used by IBM(IBM MapReduce Tools for Eclipse), Amazon,Powerset (which was acquired by Microsoft recently),Last.fm ... Read more More hadoop in data intensive scalable computing.Related projects Mahout (machine learning libraries),tashi (cluster management system).


So it was worth as I was able to attend these sessions .... Thanks to twnicnling.org