Better Search using Solr and Lucene

Solr is an open source enterprise search server based on the Lucene Java search library, with XML/HTTP and JSON APIs, hit highlighting, faceted search, caching, replication, and a web administration interface. It runs in a Java servlet container such as Apache Tomcat.

• A good tutorial for beginner: Better Search with Apache Lucene and Solr
• Tutorial at Solr HomePage 
• Slides:

Apache Solr

Lucene revisited

Lucene is an open-source full-text search library which makes it easy to add search functionality to an application or website. Want to understand Lucene in 5 minutes ? Go here. The following slide provides a quick review of Lucene.

Figure: Steps in building applications  using Lucene [Source: IBM ]

Lucene Introduction

Why Lucene ? From this DOC.

• Incremental versus batch indexing
• Data sources
• Indexing Control
• File Format
• Content Tagging
• Stop Word Processing
• Stemming
• Query Features
• Concurrency
• Non-English Support

TPTP - A Java Profiling Tool

In software engineering, program profiling, software profiling or simply profiling, a form of dynamic program analysis (as opposed to static code analysis), is the investigation of a program's behavior using information gathered as the program executes. The usual purpose of this analysis is to determine which sections of a program to optimize - to increase its overall speed, decrease its memory requirement or sometimes both.

The set of profiling tools provides software developers or testers with the ability to analyze the performance of a Java program or to gain a comprehensive understanding of the overall performance of an application. Eclipse Test and Performance Tools Platform (TPTP) is such a tool used for profiling. A good tutorial is here: Tutorial.

WEKA - Data Mining Software in Java

Weka is a collection of machine learning algorithms for data mining tasks. The algorithms can either be applied directly to a dataset or called from your own Java code. Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. It is also well-suited for developing new machine learning schemes.

HtmlCleaner - A HTML parser in JAVA

I wanted to parse webpages of a website. At first I looked at the pages' design and guessed that the HTML pages were well-formed. However, DOM parser couldn't parse the pages and informed the pages were not well-formed. More closer look revealed that some of the tags were not closed.

My next step was to search tools that  facilitate parsing of HTML pages using JAVA. I found that  a number of HTML Parsers are available to do so. Among them I chose HtmlCleaner, a tool that can CLEAN HTML web pages and can give us the DOM document. Since the pages contains Nepali characters, I must use UTF-8 encoding. Fortunately, HtmlCleaner has that capacity.

The website of HtmlCleaner doesn't show a complete sample example. However, a user has posted a sample program ( given in this URL ) that really helped me to start HTML parsing.

KompoZer - Dreamweaver like tool

I was looking for an open source tool that works similar to dreamweaver. I found KompoZer. Its official website says :

KompoZer is a complete web authoring system that combines web file management and easy-to-use WYSIWYG web page editing. KompoZer is designed to be extremely easy to use, making it ideal for non-technical computer users who want to create an attractive, professional-looking web site without needing to know HTML or web coding.

Though I haven't used it much, my first encounter with this tool was impressive.

Using Multiple Search Engines

I just found a useful website. It is really useful because it saves user's time by presenting the results of a query from two different search engines. In other words, with same input effort, one can get results from two search engines. As an example, if I want to search NEPAL in Google, I would be happy if I can get the results for NEPAL from other search engine e.g. Yahoo, on side-by-side. This would be much interesting if your screen is big enough.

Happy browsing !!

Using TreeTagger

I recently used TreeTagger to get the part-of-speech (POS) of English and French texts. As mentioned in its original website [1], TreeTagger is a language independent tool used for annotating text with part-of-speech and lemma information.

Installation is pretty easy. One just needs to follow the instructions given in the website. After the installation it will tell you something like :

You should add /home/nobal/TreeTagger/cmd and /home/nobal/TreeTagger/bin to the command search path.

And here is how you can add path (in ubuntu) :

sudo gedit /etc/bash.bashrc
at the end of file

PATH=$PATH:~/TreeTagger/bin:~/TreeTagger/cmd
export PATH

Don't forget to restart the terminal to get the effects. To verify, use this command:

echo $PATH

External Links:
[1]. TreeTagger

Ontology Search Engines

I was looking for an ontology. I found that there exist many ontology search engines. Beauty of these search engines is that they only search ontologies unlike Google, and Yahoo which are general web search engines.

 

Why do we need ontology search engines?

They help to find the suitable ontologies for given user requirements so that reuse of knowledge bases can be made.

Examples

• SWOOGLE: An ontology search engine.
• SCARLET: Discovering relations between two concepts.
• WATSON: Search Ontology and Semantically Related Documents