Implementing Full-Text Search

In etchblok-test-api, full-text search is powered by an in-memory inverted index that maps lowercase tokens to bookmark IDs. This tutorial walks you through initializing the search index, indexing your first bookmarks, and executing ranked search queries.

Prerequisites

To follow this tutorial, you need to be familiar with the core data entities in etchblok-test-api:

• BookmarkRepository: The storage layer that SearchIndex uses to fetch full bookmark objects.
• Bookmark: The data model containing the title and description fields that are indexed.

Step 1: Initializing the SearchIndex

The SearchIndex requires a BookmarkRepository instance during initialization. When you create a new SearchIndex, it automatically scans the repository and builds an initial index of all existing bookmarks.

from app.db.repository import BookmarkRepository
from app.services.search_service import SearchIndex

# Initialize the repository
repo = BookmarkRepository()

# Initialize the search index with the repository
# This calls _rebuild() internally to index existing data
search_index = SearchIndex(repo)

The SearchIndex constructor triggers a full rebuild by calling repo.list_bookmarks(page=1, per_page=10000). This ensures that the in-memory index is immediately ready for queries based on the current state of the repository.

Step 2: Indexing New Bookmarks

While the index builds itself on startup, you must manually update it when new bookmarks are created or existing ones are modified. The index_bookmark method handles both cases by first removing any old entries for the bookmark ID and then re-tokenizing the title and description.

from app.models.bookmark import Bookmark

# Create a new bookmark
new_bookmark = Bookmark(
    url="https://python.org",
    title="Python Programming Language",
    description="A powerful language for web development and data science."
)

# Persist to repository
repo.save_bookmark(new_bookmark)

# Add to the search index
search_index.index_bookmark(new_bookmark)

When index_bookmark is called, etchblok-test-api performs the following:

1. Tokenization: Splits the title and description into lowercase words.
2. Stop-word Removal: Filters out common words (like "a", "the", "is") that don't add search value.
3. Mapping: Adds the bookmark's ID to the set of IDs associated with each token in the internal _index dictionary.

Step 3: Executing a Search Query

You can query the index using the search method. It returns a list of Bookmark objects, ordered by relevance.

# Execute a search query
results = search_index.search("python web", limit=5)

for bookmark in results:
    print(f"Match: {bookmark.title} (ID: {bookmark.id})")

The search process in etchblok-test-api follows these rules:

• AND Logic: All tokens in your query must be present in a bookmark for it to be a match. If you search for "python web", only bookmarks containing both "python" and "web" are returned.
• Ranking: Results are ranked by the total number of times the query tokens appear in the bookmark's title and description combined.
• Limit: The limit parameter (defaulting to 20) restricts the number of returned results.

Step 4: Integrating with BookmarkService

In a production scenario within etchblok-test-api, you typically don't interact with SearchIndex directly. Instead, you use BookmarkService, which orchestrates the repository and the index to ensure they stay in sync.

The following example shows how BookmarkService wraps these operations:

from app.services.bookmark_service import BookmarkService

# BookmarkService is a singleton that manages SearchIndex internally
service = BookmarkService()

# Creating a bookmark via the service automatically indexes it
bookmark, error = service.create_bookmark({
    "url": "https://fastapi.tiangolo.com",
    "title": "FastAPI Framework",
    "description": "High performance, easy to learn, fast to code, ready for production"
})

# Searching via the service
search_results = service.search("fastapi performance")

By using BookmarkService.create_bookmark or BookmarkService.update_bookmark, etchblok-test-api ensures that every data mutation is reflected in the search index immediately, preventing the index from becoming stale.

Summary of Search Behavior

Feature	Implementation Detail
Storage	In-memory defaultdict(set) mapping strings to bookmark IDs.
Matching	Case-insensitive intersection (AND) of all query tokens.
Ranking	Simple frequency count of tokens in title + description.
Updates	Incremental via index_bookmark and remove_bookmark.
Persistence	None; the index is rebuilt from the repository on every application restart.