Skip to main content

Managing Index Lifecycle

To ensure search results in etchblok-test-api remain accurate after data modifications, you must manage the lifecycle of the SearchIndex. While the BookmarkService handles most synchronization automatically, certain operations—like soft-deleting or bulk updates—require manual intervention to keep the in-memory index consistent with the BookmarkRepository.

Manually Indexing a Bookmark​

You can manually add or update a bookmark in the index using the index_bookmark method. This method extracts tokens from the bookmark's title and description and maps them to the bookmark's ID.

from app.services.search_service import SearchIndex
from app.models.bookmark import Bookmark

# Assuming repo is an instance of BookmarkRepository
search_index = SearchIndex(repository=repo)

# Create or update a bookmark in the index
bookmark = Bookmark(url="https://example.com", title="Example Site", description="A useful link")
search_index.index_bookmark(bookmark)

When index_bookmark is called, it automatically performs the following:

  1. Calls _remove_bookmark_from_index to purge any existing entries for that ID (preventing duplicates).
  2. Tokenizes the combined title and description.
  3. Adds the bookmark ID to the set of IDs associated with each token in the internal _index dictionary.

Removing a Bookmark from the Index​

To completely remove a bookmark from search results, use the remove_bookmark method. This is essential when a bookmark is hard-deleted or when you want to exclude it from search results without deleting it from the repository.

# Remove a bookmark by its ID
search_index.remove_bookmark("bookmark-id-123")

Automatic Synchronization in BookmarkService​

In etchblok-test-api, the BookmarkService acts as a facade that coordinates the repository and the search index. It automatically updates the index during standard creation and update operations:

# From app/services/bookmark_service.py

def create_bookmark(self, data: Dict[str, Any]) -> Tuple[Optional[Bookmark], Optional[str]]:
    # ... validation and persistence ...
    self._repo.save_bookmark(bookmark)
    self._search.index_bookmark(bookmark) # Automatic sync
    return bookmark, None

def update_bookmark(self, bookmark_id: str, data: Dict[str, Any]) -> Tuple[Optional[Bookmark], Optional[str]]:
    # ... retrieval and update ...
    self._repo.save_bookmark(bookmark)
    self._search.index_bookmark(bookmark) # Automatic sync
    return bookmark, None

Handling Soft Deletions (Trash)​

A critical edge case in etchblok-test-api is that BookmarkService.delete_bookmark performs a "soft delete" by moving the bookmark to the trash, but it does not automatically remove it from the SearchIndex.

If you want to ensure trashed bookmarks do not appear in search results, you must manually call remove_bookmark:

# Manual synchronization for soft-deletes
def delete_and_unindex(service: BookmarkService, bookmark_id: str):
    success = service.delete_bookmark(bookmark_id)
    if success:
        # Accessing the internal search service to purge the index
        service._search.remove_bookmark(bookmark_id)

Rebuilding the Entire Index​

The SearchIndex is entirely in-memory. It is rebuilt from scratch every time the application starts or the SearchIndex class is initialized. The _rebuild method fetches all bookmarks from the repository (up to a limit of 10,000) and indexes them.

# From app/services/search_service.py

def _rebuild(self) -> None:
    """Rebuild the entire index from the repository."""
    self._index.clear()
    # Fetches all bookmarks regardless of status (active, archived, trashed)
    all_bookmarks, _ = self._repo.list_bookmarks(page=1, per_page=10000)
    for bookmark in all_bookmarks:
        self.index_bookmark(bookmark)

Troubleshooting and Gotchas​

Search Logic (AND Strategy)​

The SearchIndex.search method uses an AND strategy. If a user searches for "Python Tutorial", the index only returns bookmarks that contain both "python" and "tutorial". If a bookmark only contains one of those tokens, it will be excluded.

Stop Words​

The indexer filters out common "stop words" (e.g., "the", "and", "is") defined in app.services.search_service._STOP_WORDS. These words are ignored during both indexing and searching. If a search query consists entirely of stop words, search() will return an empty list.

In-Memory Persistence​

Because the index is in-memory, any manual changes made directly to the SearchIndex (without updating the BookmarkRepository) will be lost if the application restarts. Always ensure the underlying data in the repository is updated alongside the index.

Trashed Bookmarks in Results​

By default, SearchIndex._rebuild indexes every bookmark in the repository, including those with a trashed status. If your search results include deleted items, verify if the index was rebuilt after the items were moved to the trash.