References & JOINs — Deep dive

How reference fields power JOINs, what the schema stores, runtime behavior, reindex rules, cascading, and many‑to‑many patterns.

Related: Joins, Schema, Cascading, Field Selection

This page explains how reference fields power JOINs, what gets stored in the schema, how queries use references at runtime, what reindexing is (and isn’t) required, and how cascading interacts with bi‑directional relationships. Examples use mock models like SearchEngine::Book and SearchEngine::Author.

Overview

• Declare associations on your model: belongs_to :author and/or has_many :books.
• Compile adds a field reference: “authors.id” for author_id in the books schema (belongs_to only; has does not emit references).
• Query uses $assoc.field</code> in filters/sorts and <code>$assoc(…) in selection.
• Single‑hop only: no multi‑hop ($a.$b.field) joins.
• Reindex rules: updates to referenced docs (e.g., Author.name) don’t require reindexing referencers; changing join keys or denormalized copies does.
• Cascade: single hop, detects A ↔ B cycles and skips those pairs.
• Reference validation: during Schema.apply! the gem resolves reference: targets via alias when present, but gracefully falls back to the physical name if no alias exists. Aliases are still recommended for blue/green deploys, yet physical‑only setups are supported.

---

Declaring references with the association DSL

class SearchEngine::Book < SearchEngine::Base
  collection :books
  attribute :id, :integer
  attribute :author_id, :string  # Reference fields must be :string or [:string]

  # books.author_id → authors.id
  # belongs_to defaults are association‑name based:
  # collection: :authors; local_key: :author_id; foreign_key: :author_id
  belongs_to :author
end

What happens at compile time:

• The schema for books includes the field author_id with reference: “authors.id”.
• Types must be compatible (author_id and authors.id consistently typed as string/int64).
• Array keys ([:string]/[:integer]) model one‑to‑many (e.g., promotion_ids → promotions.id).

You can declare a reverse association on SearchEngine::Author if you need to query from Authors to Books:

class SearchEngine::Author < SearchEngine::Base
  collection :authors
  attribute :id, :integer

  # authors.id → books.author_id (reverse direction)
  has_many :books, foreign_key: :author_id
end

These two joins are independent; declare either or both, depending on how you plan to query.

---

Querying with joins

• Select from joined collections:

SearchEngine::Book
  .joins(:authors)
  .include_fields(:id, :title, authors: [:first_name, :last_name])

Compiles to: $authors(first_name,last_name),id,title.

• Filter / sort on joined fields:

SearchEngine::Book
  .joins(:authors)
  .where(authors: { last_name: "Rowling" })     # → $authors.last_name:="Rowling"
  .order(authors: { last_name: :asc })           # → $authors.last_name:asc

Rules & guardrails:

• Call .joins(:assoc) before referencing $assoc.field.
• Only single hop ($assoc.field) is supported; deeper paths are not.
• Unknown fields raise with suggestions when attributes are declared.

---

What is stored and how references are resolved

• The model DSL compiles a field‑level reference: “<target_collection>.<foreign_key>” (or ;async variant) on the referencer’s local key for belongs_to associations. This lives in the Typesense collection schema and is used by the server to resolve JOINs at query time.

Asynchronous references

• belongs_to :author, async_ref: true encodes reference: “authors.id;async” in the schema, allowing ingestion when the authors document is not yet present.
• Values you store in the referencer (e.g., author_id) must match the target field type and value (e.g., authors.id). The engine does not maintain any separate link table.
• For arrays of keys, use [:string]/[:integer] on the local side; selection returns arrays of joined documents when applicable.

---

When do you need to reindex?

Reindex referencers only when necessary. Quick rules:

• No reindex needed when a referenced document’s non‑key attributes change.
  • Example: updating Author.name is immediately visible via joins in Book queries.
• Reindex referencers when:
  • You change the join key value on the referencer (e.g., a book’s author_id).
  • You store denormalized copies of referenced fields inside the referencer (e.g., author_name on books for query_by).
  • You change schema in ways that affect join keys/field types.

Notes:

• Schema.apply! (blue/green) handles reindexing for the collection you are applying. Other collections don’t need reindex solely because of alias swaps; joins are resolved by the server using the latest data in the referenced collection.

---

Cascading and bi‑directional joins

The engine includes a cascade helper that discovers references (from live Typesense schemas or the compiled registry) and triggers reindexing of immediate referencers when a referenced document is updated.

• Single hop only; no transitive chaining.
• Cycle guard: immediate A ↔ B cycles are detected and skipped to avoid ping‑pong.
• Partial vs full: when safe (ActiveRecord source, no custom Partitioner), the engine performs a targeted partial rebuild of the referencer using the foreign key; otherwise it falls back to a full rebuild for that referencer.

Manual targeted rebuild example (use your actual key names):

# After updating an Author with id 42, refresh Books that reference it by foreign key
SearchEngine::Indexer.rebuild_partition!(
  SearchEngine::Book,
  partition: { author_id: [42] }
)

Bi‑directional joins are safe. Cascade will skip the immediate pair to prevent loops; you can still manually trigger a targeted rebuild as shown above when needed.

---

Many‑to‑many pattern

Model a bridge collection with two references (one to each side). Queries can join through the bridge in a single hop from the base to the bridge. Multi‑hop (base → bridge → other) is not supported by the JOINs DSL; denormalize or run separate searches when you need multi‑hop.

class SearchEngine::BookAuthor < SearchEngine::Base
  collection :book_authors
  attribute :book_id, :integer
  attribute :author_id, :integer

  join :books,   collection: :books,   local_key: :book_id,   foreign_key: :id
  join :authors, collection: :authors, local_key: :author_id, foreign_key: :id
end

# From books, enrich with authors via the bridge in one hop (base → bridge)
SearchEngine::Book
  .joins(:book_authors)
  .include_fields(:id, :title, book_authors: [:author_id])

---

Best practices

• Declare joins only where you actually need to query across collections.
• Prefer selection with minimal $assoc(…) fields to reduce payload and speed hydration.
• Avoid denormalizing referenced fields unless you need them for query_by/ranking; denormalization increases reindexing needs.
• Keep join keys typed consistently on both sides (int64 ↔ int64, string ↔ string).

---

Troubleshooting

• Unknown association: declare it via join :name, collection:, local_key:, foreign_key: on the base model.
• Join not applied: call .joins(:assoc) before selecting or filtering on $assoc.field.
• Unknown joined field: verify the target collection’s attributes; suggestions are provided.
• Cycle skipped: expected with A ↔ B; use a manual targeted rebuild when you truly need to refresh referencers.

Backlinks: Joins · Cascading