Relevance and Tokenization Strategy

The SearchIndex class in app.services.search_service provides a lightweight, in-memory full-text search capability for bookmarks. It is designed to handle small to medium datasets by maintaining an inverted index that maps specific keywords (tokens) to bookmark identifiers.

Tokenization and Normalization

The search process begins with tokenization, which transforms raw text from bookmark titles and descriptions into a standardized format. The _tokenize method performs three primary actions:

1. Case Normalization: All text is converted to lowercase to ensure search is case-insensitive.
2. Regex Splitting: The _TOKEN_RE regular expression ([a-z0-9]+) identifies alphanumeric sequences, effectively stripping punctuation and special characters.
3. Stop-word Filtering: Common English words that carry little semantic value (e.g., "the", "and", "is") are removed using a hardcoded _STOP_WORDS set.

# app/services/search_service.py

_STOP_WORDS: Set[str] = {"the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "for", "is", "it"}
_TOKEN_RE = re.compile(r"[a-z0-9]+")

def _tokenize(self, text: str) -> List[str]:
    """Split text into lowercase tokens, removing stop words."""
    tokens = _TOKEN_RE.findall(text.lower())
    return [t for t in tokens if t not in _STOP_WORDS]

This strategy ensures that a search for "The Python Guide" is processed as the tokens ['python', 'guide'], matching bookmarks regardless of capitalization or surrounding punctuation.

Search Matching Strategy

The SearchIndex.search method implements a strict AND matching strategy. When a user provides a multi-word query, the engine requires every token in the query to be present in the bookmark's indexed metadata for it to appear in the results.

This is achieved through set intersection of the candidate IDs associated with each token:

# app/services/search_service.py

candidate_ids: Set[str] = self._index.get(tokens[0], set()).copy()
for token in tokens[1:]:
    candidate_ids &= self._index.get(token, set())

If a query contains a token that does not exist in the index, the intersection immediately results in an empty set, returning no results. This approach prioritizes precision over recall, ensuring that results are strictly relevant to the entire query string.

Relevance and Scoring

Once a set of candidate bookmarks is identified, etchblok-test-api ranks them using a frequency-based scoring algorithm. The _rank_results method calculates a score for each bookmark based on how many times the query tokens appear in the combined title and description.

# app/services/search_service.py

@staticmethod
def _rank_results(bookmarks: List[Bookmark], tokens: List[str]) -> List[Bookmark]:
    """Rank results by number of token occurrences in title + description."""
    def score(b: Bookmark) -> int:
        text = f"{b.title} {b.description}".lower()
        return sum(text.count(t) for t in tokens)

    return sorted(bookmarks, key=score, reverse=True)

This scoring mechanism ensures that bookmarks mentioning a keyword multiple times (e.g., a bookmark about "Python" that uses the word "Python" in both the title and description) appear higher in the search results than those with only a single occurrence.

Implementation Trade-offs

The SearchIndex implementation makes several design choices that favor simplicity and speed for small datasets at the cost of scalability:

• In-Memory Persistence: The index is entirely volatile. It is rebuilt from the BookmarkRepository every time the application starts via the _rebuild method. While this avoids complex disk-based index management, it increases startup time as the number of bookmarks grows.
• Incremental Updates: The index is updated in real-time when bookmarks are added or modified. However, the _remove_bookmark_from_index method performs a full scan of all keys in the index to remove a bookmark ID. This operation is $O(N)$ relative to the number of unique tokens in the entire index, which may become a bottleneck in environments with high write volume and a large vocabulary.
• Static Stop-words: The stop-word list is hardcoded and cannot be configured without modifying the source code, which may limit the engine's effectiveness for non-English content or specialized technical vocabularies.