DeeAnalyzer

DeeAnalyzer — Primary gateway for data indexing

Synopsis

#include <dee.h>

struct              DeeAnalyzer;
struct              DeeAnalyzerClass;
gchar *             (*DeeCollatorFunc)                  (const gchar *input,
                                                         gpointer data);
void                (*DeeTermFilterFunc)                (DeeTermList *terms_in,
                                                         DeeTermList *terms_out,
                                                         gpointer filter_data);
void                dee_analyzer_add_term_filter        (DeeAnalyzer *self,
                                                         DeeTermFilterFunc filter_func,
                                                         gpointer filter_data,
                                                         GDestroyNotify filter_destroy);
void                dee_analyzer_analyze                (DeeAnalyzer *self,
                                                         const gchar *data,
                                                         DeeTermList *terms_out,
                                                         DeeTermList *colkeys_out);
gint                dee_analyzer_collate_cmp            (DeeAnalyzer *self,
                                                         const gchar *key1,
                                                         const gchar *key2);
gint                dee_analyzer_collate_cmp_func       (const gchar *key1,
                                                         const gchar *key2,
                                                         gpointer analyzer);
gchar *             dee_analyzer_collate_key            (DeeAnalyzer *self,
                                                         const gchar *data);
DeeAnalyzer *       dee_analyzer_new                    (void);
void                dee_analyzer_tokenize               (DeeAnalyzer *self,
                                                         const gchar *data,
                                                         DeeTermList *terms_out);

Object Hierarchy

  GObject
   +----DeeAnalyzer
         +----DeeTextAnalyzer

Description

A DeeAnalyzer takes a text stream, splits it into tokens, and runs the tokens through a series of filtering steps. Optionally outputs collation keys for the terms.

One of the important use cases of analyzers in Dee is as vessel for the indexing logic for creating a DeeIndex from a DeeModel.

The recommended way to implement your own custom analyzers are by either adding term filters to a DeeAnalyzer or DeeTextAnalyzer instance with dee_analyzer_add_term_filter() and/or derive your own subclass that overrides the dee_analyzer_tokenize() method. Should you have very special requirements it is possible to reimplement all aspects of the analyzer class though.

Details

struct DeeAnalyzer

struct DeeAnalyzer;

All fields in the DeeAnalyzer structure are private and should never be accessed directly

struct DeeAnalyzerClass

struct DeeAnalyzerClass {
};

DeeCollatorFunc ()

gchar *             (*DeeCollatorFunc)                  (const gchar *input,
                                                         gpointer data);

A collator takes an input string, most often a term produced from a DeeAnalyzer, and outputs a collation key.

input :

 The string to produce a collation key for

data :

 User data set when registering the collator. [closure]

Returns :

 The collation key. Free with g_free() when done using it. [transfer full]

DeeTermFilterFunc ()

void                (*DeeTermFilterFunc)                (DeeTermList *terms_in,
                                                         DeeTermList *terms_out,
                                                         gpointer filter_data);

A term filter takes a list of terms and runs it through a filtering and/or set of transformations and stores the output in a DeeTermList.

You can register term filters on a DeeAnalyzer with dee_analyzer_add_term_filter().

terms_in :

 A DeeTermList with the terms to filter

terms_out :

 A DeeTermList to write the filtered terms to

filter_data :

 User data set when registering the filter. [closure]

dee_analyzer_add_term_filter ()

void                dee_analyzer_add_term_filter        (DeeAnalyzer *self,
                                                         DeeTermFilterFunc filter_func,
                                                         gpointer filter_data,
                                                         GDestroyNotify filter_destroy);

Register a DeeTermFilterFunc to be called whenever dee_analyzer_analyze() is called.

Term filters can be used to normalize, add, or remove terms from an input data stream.

self :

 The analyzer to add a term filter to

filter_func :

 Function to call. [scope notified]

filter_data :

 Data to pass to filter_func when it is invoked. [closure]

filter_destroy :

 Called on filter_data when the DeeAnalyzer owning the filter is destroyed. [allow-none]

dee_analyzer_analyze ()

void                dee_analyzer_analyze                (DeeAnalyzer *self,
                                                         const gchar *data,
                                                         DeeTermList *terms_out,
                                                         DeeTermList *colkeys_out);

Extract terms and or collation keys from some input data (which is normally, but not necessarily, a UTF-8 string).

The terms and corresponding collation keys will be written in order to the provided DeeTermLists.

Implementation notes for subclasses: The analysis process must call dee_analyzer_tokenize() and run the tokens through all term filters added with dee_analyzer_add_term_filter(). Collation keys must be generated with dee_analyzer_collate_key().

self :

 The analyzer to use

data :

 The input data to analyze

terms_out :

 A DeeTermList to place the generated terms in. If NULL to terms are generated. [allow-none]

colkeys_out :

 A DeeTermList to place generated collation keys in. If NULL no collation keys are generated. [allow-none]

dee_analyzer_collate_cmp ()

gint                dee_analyzer_collate_cmp            (DeeAnalyzer *self,
                                                         const gchar *key1,
                                                         const gchar *key2);

Compare collation keys generated by dee_analyzer_collate_key() with similar semantics as strcmp(). See also dee_analyzer_collate_cmp_func() if you need a version of this function that works as a GCompareDataFunc.

The default implementation in DeeAnalyzer just uses strcmp().

self :

 The analyzer to use when comparing collation keys

key1 :

 The first collation key to compare

key2 :

 The second collation key to compare

Returns :

 -1, 0 or 1, if key1 is <, == or > than key2.

dee_analyzer_collate_cmp_func ()

gint                dee_analyzer_collate_cmp_func       (const gchar *key1,
                                                         const gchar *key2,
                                                         gpointer analyzer);

A GCompareDataFunc using a DeeAnalyzer to compare the keys. This is just a convenience wrapper around dee_analyzer_collate_cmp().

key1 :

 The first key to compare

key2 :

 The second key to compare

analyzer :

 The DeeAnalyzer to use for the comparison

Returns :

 -1, 0 or 1, if key1 is <, == or > than key2.

dee_analyzer_collate_key ()

gchar *             dee_analyzer_collate_key            (DeeAnalyzer *self,
                                                         const gchar *data);

Generate a collation key for a set of input data (usually a UTF-8 string passed through tokenization and term filters of the analyzer).

The default implementation just calls g_strdup().

self :

 The analyzer to generate a collation key with

data :

 The input data to generate a collation key for

Returns :

 A newly allocated collation key. Use dee_analyzer_collate_cmp() or dee_analyzer_collate_cmp_func() to compare collation keys. Free with g_free().

dee_analyzer_new ()

DeeAnalyzer *       dee_analyzer_new                    (void);

dee_analyzer_tokenize ()

void                dee_analyzer_tokenize               (DeeAnalyzer *self,
                                                         const gchar *data,
                                                         DeeTermList *terms_out);

Tokenize some input data (which is normally, but not necessarily, a UTF-8 string).

Tokenization splits the input data into constituents (in most cases words), but does not run it through any of the term filters set for the analyzer. It is undefined if the tokenization process itself does any normalization.

self :

 The analyzer to use

data :

 The input data to analyze

terms_out :

 A DeeTermList to place the generated tokens in.
Generated by GTK-Doc V1.18