Better Search using Solr and Lucene

Solr is an open source enterprise search server based on the Lucene Java search library, with XML/HTTP and JSON APIs, hit highlighting, faceted search, caching, replication, and a web administration interface. It runs in a Java servlet container such as Apache Tomcat.

• A good tutorial for beginner: Better Search with Apache Lucene and Solr
• Tutorial at Solr HomePage 
• Slides:

Apache Solr

Lucene revisited

Lucene is an open-source full-text search library which makes it easy to add search functionality to an application or website. Want to understand Lucene in 5 minutes ? Go here. The following slide provides a quick review of Lucene.

Figure: Steps in building applications  using Lucene [Source: IBM ]

Lucene Introduction

Why Lucene ? From this DOC.

• Incremental versus batch indexing
• Data sources
• Indexing Control
• File Format
• Content Tagging
• Stop Word Processing
• Stemming
• Query Features
• Concurrency
• Non-English Support

HTML5 - The Future of the Web

This post provides a quick introduction to  HTML5- the future of the web.

1. VIDEO

2. SLIDES:

Regarding Twitter

All the contents in this blog posts are taken from this paper.

Twitter.com is a online social network used by millions of people around the world to stay connected to their friends, family members and coworkers through their computers and mobile phones. The interface allows users to post short messages (up to 140 characters) that can be read by any other Twitter user. 

Users declare the people they are interested in following, in which case they get notified when that person has posted a new message. A user who is being followed by another user does not necessarily have to reciprocate by following them back, which makes the links of the Twitter social network directed.

Twitter users are able to post direct and indirect updates. Direct posts are used when a user aims her update to a specific person, whereas indirect updates are used when the update is meant for anyone that cares to read it.

Even though direct updates are used to communicate directly with a specific person, they are public and anyone can see them.

FRIEND : Here, a user’s friend is a person whom the user has directed at least two posts to.

Research Findings :

• the number of posts initially increases as the number of followers increases but it eventually saturates.
• the number of posts increases as the number of friends increases
• the users who receive attention from many people will post more often than users who receive little attention.
• in order to predict how active a Twitter user is, the number of friends is a more accurate signal than the number of his followers.
• most users have a very small number of friends compared to the number of followees they declared.
• the cost of declaring a new followee is very low compared to the cost of maintaining a friends (i.e. exchanging directed messages with other users). Hence, the number of people a user actually communicates with eventually stops increasing while the number of followees can continue to grow indefinitely.
• users with more followers and friends will be more active at posting than those with a small number of followers and friends.
• a link between any two people does not necessarily imply an interaction between them. in the case of Twitter, most of the links declared within Twitter were meaningless from an interaction point of view. Thus the need to find the hidden social network; the one that matters when trying to rely on word of mouth to spread an idea, a belief, or a trend.

Conclusion:

In conclusion, even when using a very weak definition of “friend” (i.e. anyone who a user has directed a post to at least twice) we find that Twitter users have a very small number of friends compared to the number of followers and followees they declare. This implies the existence of two different networks: a very dense one made up of followers and followees, and a sparser and simpler network of actual friends. The latter proves to be a more influential network in driving Twitter usage since users with many actual friends tend to post more updates than users with few actual friends. On the other hand, users with many followers or followees post updates more infrequently than those with few followers or followees.

Real-time web search

I loved this article because it provided me an information about real-time web searching which is at its infancy. Real-time web searching means searching the real time content. For example, if a great politician dies, people generate content exponentially. Providing relevant information in real time is not so easy. Here I'm listing some of the points that I liked in the article.

• Now a delay of minutes on a breaking news story is unacceptable
• Real-time search starts by determining that something important is happening in, well, real time.
• Real-time search today is in its infancy, but it's the next stage in the evolution of Internet search.
• RT Searching should address how can the explosion of instant content produced by news organizations, blogs, and social-media users be organized so that results can be provided instantly
• what is "real-time" content?: -it centers on the concept of microblogging, or instant publishing of content to the open Web from social-media services. But in practice, "real-time search is still primarily Twitter search
• two components to real-time information: the actual content of the status update or post, and the link that is being shared within that update.
• Why web search providers want to buy Twitter's 'Firehouse' ?... Why spend the money? It's simply too difficult to crawl Twitter the way traditional search engines crawl the Web. All three major search engines (Y,G,B) at this point have inked deals to have Twitter push its content directly to them, saving those companies (and Twitter) time, energy, and money.
• deadlines are dead in the real-time world.
• So if search engines are to remain relevant themselves, they'll need to make sense of this content. And unless social-media networks are able to make their content discoverable, they won't turn into the types of content-discovery engines that their public-relations people like to imagine are already here.
• Expect the importance of real-time search to only grow over the next several years. For example, Yahoo's search deal with Microsoft does not include real-time indexing and ranking efforts, as the company believes that it's too important to give away.

Interesting Links:

• Oneriot.com - Assumes that the content based on on the premise that the link being shared within the status update is more relevant than the message itself.
• Wowd.com - An example search engine of real-time web searching

XML Namespace: Attributes are a little different

An attribute can appear in a different namespace than the element that contains it. For example, <movie:title xml:lang="fr"> has an attribute that is not from the movie namespace. If an attribute name has a prefix, its name is in the namespace indicated by the prefix. However, if an attribute name has no prefix, it has no namespace. This is true even when the default namespace has been assigned. The W3C Namespaces in XML Recommendation makes that point with this example:

<x xmlns="http://www.w3.org" xmlns:n1="http://www.w3.org"> <good a="1" n1:a="2" /> </x>

The elements are affected by the declaration of a URI for the default namespace. That is, both x and good are associated with the URI "http://www.w3.org" because it's the default namespace. The attribute n1:a is also associated with that namespace, due to its use of the n1 prefix, which is associated with the same URI. There is no conflict that the a attribute is being declared twice, because while n1:a is in the http://www.w3.org namespace, the unprefixed a is not; the latter is not in any namespace.

Reference:

Copied from XML Namespace
Another Interesting tutorial for XML NS: Here

Dependency Trees

A dependency tree is a graphical representation of a sentence parsed using a dependency grammar. The nodes in the tree correspond to words in the sentence being parsed (and sometimes to special synthesized nodes). The arcs correspond to dependency relations between a "head" word, at the upper end of an arc, and the dependent words at the lower ends of the arcs connected to the head word. The grammatical relations between head and dependent words are such things as subject, object, modifier, etc.

Ontology learning

Ontology learning also known as ontology extraction, ontology generation or ontology acquisition is a semi-automatic way of information extraction which is used to build an ontology from scratch (finding concepts and their relations), enriching or adopting an existing ontology.

Federated Search

Federated search is the simultaneous search of multiple online databases or web resources and is an emerging feature of automated, web-based library and information retrieval systems. It is also often referred to as a portal or a federated search engine.

Named Entity Recognization

Named Entity Recognition (NER) is also known as entity extraction and entity recognition. NER, a subtask of information extraction, is a process of finding mentions of specified things in the given text. In other words, it seeks to locate and classify atomic elements in text into predefined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc.

Most research on NER systems has been structured as taking an unannotated block of text, such as this one:
Jim bought 300 shares of Acme Corp. in 2006.

And producing an annotated block of text, such as this one:

<ENAMEX TYPE="PERSON">Jim</ENAMEX> bought <NUMEX TYPE="QUANTITY">300</NUMEX> shares of <ENAMEX TYPE="ORGANIZATION">Acme Corp.</ENAMEX> in <TIMEX TYPE="DATE">2006</TIMEX>

In this example, the annotations have been done using so-called ENAMEX tags that were developed for the Message Understanding Conference in the 1990s.

• Performance: state-of-the-art NER systems for English produce near-human performance.
• Tools: Wikipedia lists a number of open source tools such as MALLET
  

Design Patterns

Design patterns are the repeating patterns of solving problems in software. A nice book called "The Design Patterns, Java Companion" [ available here ] by J. W. Cooper explains design patterns in very simple words. He says that when we tell a colleague how we accomplished a tricky bit of programming so he doesn’t have to recreate it from scratch. We simply recognize effective ways for objects to communicate while maintaining their own separate existences.

He has collected some useful definitions of design patterns:

• “Design patterns are recurring solutions to design problems you see over
• “Design patterns constitute a set of rules describing how to accomplish certain tasks in the realm of software development.”
• “Design patterns focus more on reuse of recurring architectural design themes, while frameworks focus on detailed design… and implementation.”.
• “A pattern addresses a recurring design problem that arises in specific design situations and presents a solution to it”
  
• “Patterns identify and specify abstractions that are above the level of single classes and instances, or of components.”

But while it is helpful to draw analogies to architecture, cabinet making and logic, design patterns are not just about the design of objects, but about the communication between objects. In fact, we sometimes think of them as communication patterns. It is the design of simple, but elegant, methods of communication that makes many design patterns so important.

Page Rank Algorithm

Google uses Page Rank Algorithm (PRA) to rank web pages so that finding needle in web's haystack can be accomplished. Here is a good tutorial that explains how it finds our pages in the web.

How Google Finds Your Needle in the Web's Haystack

Fun with mathematics II

We can do multiplication by means of visual ways. I have found both video and article that shows how one can compute the multiplication of ANY DIGIT NUMBERS using visual method.

A. Video
I am hundred percent sure that you will enjoy this video. Watch it first and then we discuss the principle behind this approach.

B. Article

Note: I have copied the following paragraphs and a picture from [1]. If you don't understand here, please go to that site.

Example: Multiply 22 by 13.

Note: This figure is taken from [1]

Draw 2 lines slanted upward to the right, and then move downward to the right a short distance and draw another 2 lines upward to the right (see the magenta lines in Figure 1). Then draw 1 line slanted downward to the right, and then move upward to the right a short distance and draw another 3 lines slanted downward to the right (the cyan lines in Figure 1).

Now count up the number of intersection points in each corner of the figure. The number of intersection points at left (green-shaded region) will be the first digit of the answer. Sum the number of intersection points at the top and bottom of the square (in the blue-shaded region); this will be the middle digit of the answer. The number of intersection points at right (in the yellow-shaded region) will be the last digit of the answer.

This will work to multiply any two two-digit numbers, but if any of the green, blue, gold sums have 10 or more points in them, be sure to carry the tens digit to the left, just as you would if you were adding.

C. Understanding the LOGIC
Below, I've described the way we did the multiplication in our school and if you have noticed, the same thing is happening in the visual method as well. Here it goes:

Ex1
   2 2
x 1 3
-------
   6  6
2 2  x
-------------
2 8 6

Ex2:
1 5 6
3 5 8
---------------------------------------
           8    40   48
     5  25    30   x x
 3 15 18    xx    x x
-------------------------
 5   5  8      4     8 [ while adding , carry should be propagated towards left ]

We know that two non parallel lines always meet exactly at a point. Note that when a number of lines (representing one digit number, e.g. five lines for 5 ) crosses  a number of other lines ( which represents another one digit number), then the number of  points formed by the crossings is equal to their product.

I understood the logic by referring back to my ways of doing the calculation. It may not be clear to you by my explanation. Better visit the website and read the article in such a case.

References:

1. Su, Francis E., et al. "Squaring Quickly." Mudd Math Fun Facts. 

Levenshtein distance

Levenshtein distance, also known as edit distance, is a metric that measures the amount of differences between two given sequences. Wikipedia says the Levenshtein distance between two strings is given by the minimum number of operations needed to transform one string into the other, where an operation is an insertion, deletion, or substitution of a single character.

Usage: It is often used in applications that need to determine how similar, or different, two strings are, such as spell checkers.

Examples:
The Levenshtein distance between "kitten" and "sitting" is 3, since the following three edits change one into the other, and there is no way to do it with fewer than three edits:

1. kitten → sitten (substitution of 's' for 'k')
2. sitten → sittin (substitution of 'i' for 'e')
3. sittin → sitting (insert 'g' at the end).