Skip to main content

Overview of the Search System

The search system in this project is a lightweight, in-memory full-text search engine implemented via the SearchIndex class in app/services/search_service.py. It provides efficient retrieval of bookmarks by mapping text tokens to bookmark identifiers, allowing for fast keyword-based lookups without querying the underlying database for every search operation.

Inverted Index Architecture​

The core of the search system is an inverted index. Instead of scanning every bookmark to see if it contains a keyword, the SearchIndex maintains a dictionary (self._index) where keys are individual words (tokens) and values are sets of bookmark IDs that contain those words.

# Internal structure of the index in app/services/search_service.py
self._index: Dict[str, Set[str]] = defaultdict(set)

When a search is performed, the system looks up the sets of IDs for each token in the query and finds their intersection, ensuring that only bookmarks containing all search terms are returned.

Tokenization and Filtering​

Before text is added to the index or used in a query, it passes through the _tokenize method. This process ensures that searches are case-insensitive and ignore common "stop words" that don't add semantic value.

  1. Normalization: Text is converted to lowercase.
  2. Regex Splitting: The _TOKEN_RE (re.compile(r"[a-z0-9]+")) extracts alphanumeric sequences.
  3. Stop Word Removal: Words defined in _STOP_WORDS (such as "the", "and", "is") are discarded.
# app/services/search_service.py
_STOP_WORDS: Set[str] = {"the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "for", "is", "it"}

def _tokenize(self, text: str) -> List[str]:
    """Split text into lowercase tokens, removing stop words."""
    tokens = _TOKEN_RE.findall(text.lower())
    return [t for t in tokens if t not in _STOP_WORDS]

Index Lifecycle and Synchronization​

The SearchIndex is designed to be a live representation of the bookmark repository. It handles both initial bootstrapping and incremental updates.

Bootstrapping on Startup​

When the application starts, the BookmarkService initializes the SearchIndex. The index immediately calls _rebuild(), which fetches all existing bookmarks from the BookmarkRepository and populates the index.

def _rebuild(self) -> None:
    """Rebuild the entire index from the repository."""
    self._index.clear()
    all_bookmarks, _ = self._repo.list_bookmarks(page=1, per_page=10000)
    for bookmark in all_bookmarks:
        self.index_bookmark(bookmark)

Incremental Updates​

To keep the index synchronized with the database, the BookmarkService triggers index updates during CRUD operations:

  • Creation/Update: When create_bookmark or update_bookmark is called, the service invokes index_bookmark(bookmark). This method first removes any existing entries for that ID and then re-indexes the title and description.
  • Deletion: When a bookmark is deleted, remove_bookmark(bookmark_id) is called to purge its ID from all token sets in the index.

Search Logic and Ranking​

The search method implements an "AND" logic for multi-token queries. If a user searches for "python tutorial", the system retrieves the set of IDs for "python" and the set for "tutorial", then performs a set intersection.

Ranking Results​

Once matching bookmarks are identified, they are ranked using the _rank_results static method. The ranking is based on simple term frequency: the more times the search tokens appear in the bookmark's title and description, the higher the score.

@staticmethod
def _rank_results(bookmarks: List[Bookmark], tokens: List[str]) -> List[Bookmark]:
    """Rank results by number of token occurrences in title + description."""
    def score(b: Bookmark) -> int:
        text = f"{b.title} {b.description}".lower()
        return sum(text.count(t) for t in tokens)

    return sorted(bookmarks, key=score, reverse=True)

Integration with the API​

The search system is exposed via the /api/bookmarks/search endpoint. The route handler in app/routes/bookmarks.py extracts the query string q and an optional limit, passing them to the BookmarkService, which delegates the actual work to the SearchIndex.

# Example usage in app/routes/bookmarks.py
@bookmarks_bp.route("/search", methods=["GET"])
def search_bookmarks():
    query = request.args.get("q", "")
    limit = request.args.get("limit", 20, type=int)
    results = _service.search(query, limit=limit)
    return jsonify({"results": [b.to_dict() for b in results], "count": len(results)})

Performance Considerations​

  • In-Memory: The index resides entirely in RAM. While extremely fast for retrieval, it must be rebuilt if the application restarts.
  • Removal Overhead: The _remove_bookmark_from_index method performs a full scan of the index dictionary to find and remove a bookmark ID. As the number of unique tokens grows, this operation becomes more expensive.
  • Scalability: This implementation is optimized for small to medium datasets. For very large collections of bookmarks, the memory footprint and rebuild time would necessitate a dedicated search engine like Elasticsearch or Typesense.