Understanding the Search Architecture

When you need to find a specific bookmark but can only remember a few keywords from its description, etchblok-test-api uses an in-memory search architecture to provide fast, full-text results. This system avoids the overhead of complex external search engines by maintaining a local inverted index that maps keywords directly to bookmark IDs.

The In-Memory Inverted Index

The core of the search functionality is the SearchIndex class in app/services/search_service.py. It maintains an internal dictionary called _index where keys are unique tokens (words) and values are sets of bookmark IDs containing those tokens.

# Internal structure of SearchIndex._index
{
    "python": {"uuid-1", "uuid-4"},
    "tutorial": {"uuid-1", "uuid-2"},
    "flask": {"uuid-4"}
}

When you initialize the BookmarkService (which is a singleton), it creates an instance of SearchIndex. The index immediately performs a full rebuild by fetching all bookmarks from the BookmarkRepository via the _rebuild() method:

def _rebuild(self) -> None:
    """Rebuild the entire index from the repository."""
    self._index.clear()
    all_bookmarks, _ = self._repo.list_bookmarks(page=1, per_page=10000)
    for bookmark in all_bookmarks:
        self.index_bookmark(bookmark)

Tokenization and Filtering

Before text is added to the index or used in a query, it passes through the _tokenize method. This ensures that search is case-insensitive and ignores common "noise" words that don't help distinguish results.

1. Normalization: Text is converted to lowercase.
2. Extraction: The _TOKEN_RE regex ([a-z0-9]+) extracts alphanumeric sequences.
3. Filtering: Words found in the _STOP_WORDS set (e.g., "the", "and", "is") are discarded.

_STOP_WORDS = {"the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "for", "is", "it"}

def _tokenize(self, text: str) -> List[str]:
    """Split text into lowercase tokens, removing stop words."""
    tokens = _TOKEN_RE.findall(text.lower())
    return [t for t in tokens if t not in _STOP_WORDS]

The indexer specifically targets the title and description fields of a Bookmark object. Other fields like tags or URLs are currently not indexed for search.

Search Logic and Ranking

When you perform a search via the /search endpoint, the SearchIndex.search method implements AND logic. This means a bookmark must contain all tokens from your query to be considered a match.

Internally, this is achieved by taking the intersection of the ID sets for each token:

candidate_ids: Set[str] = self._index.get(tokens[0], set()).copy()
for token in tokens[1:]:
    candidate_ids &= self._index.get(token, set())

Once the candidate bookmarks are retrieved from the repository, they are ranked using _rank_results. The ranking is based on a simple frequency score: the total number of times the query tokens appear in the bookmark's title and description.

def _rank_results(bookmarks: List[Bookmark], tokens: List[str]) -> List[Bookmark]:
    def score(b: Bookmark) -> int:
        text = f"{b.title} {b.description}".lower()
        return sum(text.count(t) for t in tokens)

    return sorted(bookmarks, key=score, reverse=True)

Maintaining Data Consistency

To ensure search results remain accurate as you add or modify bookmarks, BookmarkService performs incremental updates to the index. You don't need to manually trigger a re-index; the service handles it during standard CRUD operations.

In app/services/bookmark_service.py, methods like create_bookmark and update_bookmark call self._search.index_bookmark(bookmark) after successfully persisting changes to the database:

def update_bookmark(self, bookmark_id: str, data: Dict[str, Any]) -> Tuple[Optional[Bookmark], Optional[str]]:
    # ... update logic ...
    self._repo.save_bookmark(bookmark)
    self._search.index_bookmark(bookmark) # Incremental index update
    self._cache.invalidate(bookmark.id)
    return bookmark, None

The index_bookmark method first removes any existing entries for that bookmark ID to prevent stale data, then re-tokenizes the updated content and adds it back to the index.

Limitations to Consider

• Memory Usage: Since the index is stored entirely in RAM, it is best suited for small to medium datasets.
• Strict Matching: Because of the AND logic, a query for "Python Tutorial" will not return a bookmark that only contains "Python".
• Persistence: The index is not saved to disk. It is entirely reconstructed from the database every time the etchblok-test-api process starts.