> ## Documentation Index
> Fetch the complete documentation index at: https://nikita-shkoda.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Schema

> Compile model DSL to Typesense schema, diff against live, apply with blue/green + retention, and rollback.

Related: <a href="/projects/search-engine-for-typesense/v30/cli">CLI</a>, <a href="/projects/search-engine-for-typesense/v30/troubleshooting#schema">Troubleshooting → Schema</a>

The schema layer turns a model class (our DSL) into a Typesense-compatible schema hash and compares it to the live, currently aliased physical collection to surface drift.

```mermaid theme={null}
flowchart LR
  A[Collection class] --> B[Schema builder]
  B --> C[Compiled schema]
  C --> D[Compare with active physical]
  D --> E[Diff report]
```

## API

* <code>SearchEngine::Schema.compile(klass)</code> → returns a Typesense-compatible schema hash built from the DSL. Pure and deterministic (no network I/O).
* <code>SearchEngine::Schema.diff(klass)</code> → resolves alias → physical, fetches the live schema, and returns a structured diff plus a compact human summary.
* <code>SearchEngine::Schema.update!(klass)</code> → attempts an in-place schema patch (Typesense <code>PATCH /collections/:name</code>) when the diff only contains field additions/drops; returns <code>true</code> when the schema is already in sync or successfully patched.
* <code>SearchEngine::Schema.apply!(klass, force\_rebuild: false)</code> → blue/green lifecycle (create new physical, reindex, swap alias, retention). By default it first tries <code>update!</code> and only falls back to blue/green when incompatible changes are detected. Returns `{ logical, new_physical, previous_physical, alias_target, dropped_physicals, action: :update|:rebuild }`.
* <code>SearchEngine::Schema.rollback(klass)</code> → swap alias back to previous retained physical; returns `{ logical, new_target, previous_target }`.
* <code>SearchEngine::Schema.prune\_orphans!(client: nil, logical: nil)</code> → drop orphaned physical collections that no alias points to. Scoped to a single logical name when `logical:` is provided; scans all collections when `nil`. Returns `{ dropped: Array, kept: Array, total_scanned: Integer }`.

These APIs are documented with YARD. Keys are returned as symbols; empty/nil values are omitted.
The returned schema is deeply frozen.

## In-place schema updates (Typesense v29+)

* <code>SearchEngine::Schema.update!(klass, client: ...)</code> inspects the live diff and issues a <code>PATCH /collections/:name</code> when the changes are limited to field additions or drops. Type changes, reference changes, or collection-level option differences automatically return <code>false</code>, signalling that a full blue/green rebuild is required.
* <code>klass.update\_collection!</code> (available on every <code>SearchEngine::Base</code> subclass) is a convenience wrapper that logs console guidance and delegates to <code>Schema.update!</code>.
* <code>Schema.apply!(force\_rebuild: false)</code> now attempts an in-place update first. Pass <code>force\_rebuild: true</code> when you explicitly need to skip PATCH (for example, when you want a new physical even if only field additions are pending).

CLI tasks such as <code>bin/rails search\_engine:schema:apply\[Collection]</code> inherit the same behavior because they call <code>Schema.apply!</code> under the hood.

## Type mapping (DSL → Typesense)

* <strong>:string</strong> → <code>string</code>
* <strong>:integer</strong> → <code>int64</code> (chosen consistently for wider range)
* <strong>:float / :decimal</strong> → <code>float</code>
* <strong>:boolean</strong> → <code>bool</code>
* <strong>:time / :datetime</strong> → <code>int64</code> (epoch seconds)
* <strong>:time\_string / :datetime\_string</strong> → <code>string</code> (ISO8601 timestamps)
* Arrays like <code>\[:string]</code> → <code>string\[]</code>
* <strong>:geopoint</strong> → <code>geopoint</code> (Typesense native geographic coordinate)
* Arrays like <code>\[:geopoint]</code> → <code>geopoint\[]</code>
* <strong>:auto</strong> (regex-style field names such as `".*_facet"`) → `auto`; enables Typesense auto schema detection and wildcard ingestion. The DSL enforces that `:auto` can only be used when the attribute name looks like a regex (contains metacharacters such as `*`, `.`, etc.).

### Array empty filtering (hidden fields)

When declaring an array attribute, you can enable automatic empty filtering by adding <code>empty\_filtering: true</code>:

```ruby theme={null}
attribute :promotion_ids, [:string], empty_filtering: true
```

Behavior:

* Schema includes a hidden boolean field <code>promotion\_ids\_empty</code>.
* The mapper auto-populates it per document as: <code>promotion\_ids.nil? || promotion\_ids.empty?</code>.
* Hidden fields are not exposed via public APIs or <code>inspect</code>; they are internal.

Constraints:

* <code>empty\_filtering</code> is only valid for array types (e.g., <code>\[:string]</code>); setting it on scalars raises an error.

Query rewrite:

* <code>.where(promotion\_ids: \[])</code> → <code>promotion\_ids\_empty:=true</code>
* <code>.where.not(promotion\_ids: \[])</code> → <code>promotion\_ids\_empty:=false</code>

Joins:

* For joined filters like `.joins(:brand).where(brand: { promotion_ids: [] })`, the rewrite applies only if the joined collection has <code>attribute :promotion\_ids, \[:string], empty\_filtering: true</code> (hidden <code>\$brand.promotion\_ids\_empty</code> exists). Otherwise an empty array remains invalid.

### Unindexed embed sources

Fields declared with `index: false` are normally omitted from the compiled schema. However, when such a field is listed in an embedding's `from:` array, the compiler keeps it in the schema with Typesense-native `"index": false` and forced `"optional": true`. This lets the embedding model read the value without keyword-indexing it.

No `<name>_blank` hidden field is generated for these fields because the DSL-level `optional` is not set — the `"optional": true` in the Typesense payload is an implementation detail required by Typesense for unindexed fields.

### System field: `doc_updated_at`

* Always present on every collection. **Cannot be disabled**—the gem automatically injects this field during document creation/upsert.
* Stored in Typesense as <code>int64</code> (epoch seconds). If declared in the model DSL, its type will be coerced to <code>int64</code> at compile time to ensure consistency.
* On hydration and console output, it is converted to a <code>Time</code> in the current timezone (uses <code>Time.zone</code> when available, falling back to <code>Time</code>).
* When using instance <code>attributes</code>, <code>:doc\_updated\_at</code> is returned as a <code>Time</code> object. Unknown fields remain available under <code>:unknown\_attributes</code>.
* **Typesense limitation**: This field is required by Typesense for internal tracking. The gem enforces its presence to maintain compatibility.

## Collection options

If declared in the DSL in the future, the builder may include top-level options like <code>default\_sorting\_field</code>, <code>token\_separators</code>, <code>symbols\_to\_index</code>. Today, these are omitted to avoid noisy diffs.

### Nested fields (auto-enabled)

* When any attribute is declared with type <code>:object</code> or <code>\[:object]</code>, the schema compiler will automatically set <code>enable\_nested\_fields: true</code> at the collection level.
* This is required by Typesense to accept <code>object</code> / <code>object\[]</code> field types; otherwise the server responds with <code>400 RequestMalformed</code>.
* The option is included in <code>Schema.apply!</code> create payloads and appears under <code>collection\_options</code> in <code>Schema.diff</code>.
* If you don't need nested objects, consider flattening fields or storing JSON as a <code>:string</code>.

#### Declaring nested subfields

Declare subfields inline via the <code>nested:</code> option on the base attribute:

```ruby theme={null}
attribute :retail_prices, [:object], nested: {
  current_price: :float,
  general_price: :float,
  current_discount_percent: :float,
  current_minimum_quantity: :integer,
  price_type: :string
}
```

Multiplicity rule:

* Base <code>:object</code> → subfields are scalars (<code>float</code>, <code>int64</code>, <code>string</code>).
* Base <code>\[:object]</code> → subfields are arrays (<code>float\[]</code>, <code>int64\[]</code>, <code>string\[]</code>).

See also: Typesense docs on <code>enable\_nested\_fields</code> in <code>collections.create</code> (<code>typesense.org</code>).

## Diff shape

```text theme={null}
{
  collection: { name: String, physical: String },
  added_fields: [ { name: String, type: String }, ... ],
  removed_fields: [ { name: String, type: String }, ... ],
  changed_fields: { "field" => { "type" => [compiled, live] } },
  collection_options: { /* option => [compiled, live] */ }
}
```

* Field comparison is name-keyed and order-insensitive.
* Only changed keys appear under <code>changed\_fields</code>.
* When the live collection is missing, <code>added\_fields</code> contain all compiled fields and <code>collection\_options</code> includes <code>live: :missing</code>.

## Pretty print

The human summary includes:

* <strong>Header</strong>: logical and physical names
* <strong>+ Added fields</strong>: <code>name:type</code>
* <strong>- Removed fields</strong>: <code>name:type</code>
* <strong>\~ Changed fields</strong>: <code>field.attr compiled→live</code>
* <strong>\~ Collection options</strong>: shown only when differing

Example (no changes):

```text theme={null}
Collection: products
No changes
```

## Lifecycle (Blue/Green with retention)

```mermaid theme={null}
sequenceDiagram
  participant App
  participant TS as Typesense
  App->>TS: Create new physical (versioned)
  App->>TS: Bulk reindex into new physical
  App->>TS: Alias swap (logical -> new physical)
  App->>TS: Retention cleanup (drop old N)
```

* Physical name format: <code>"#{logical}*YYYYMMDD\_HHMMSS*###"</code> (3-digit zero-padded sequence).
* Alias equals the logical name (e.g., <code>products</code>). Swap is performed via a single upsert call, which the server handles atomically.
* Idempotent: if alias already points to the new physical, swap is a no-op.
* Reindexing is required. Provide a block to <code>apply!</code> or implement <code>klass.reindex\_all\_to(physical\_name)</code> to perform bulk import. On failure or interruption, no alias swap occurs and the new physical collection is automatically deleted to prevent orphan accumulation.

<Info>
  Creating a physical collection manually, importing into it, and calling <code>Client#upsert\_alias</code> directly will NOT trigger retention cleanup. Old physical collections remain until removed explicitly. Retention cleanup only runs as part of <code>Schema.apply!</code> (and the <code>search\_engine:schema:apply\[...]</code> task), after a successful alias swap.
</Info>

### Retention

* Global default: keep none.

```ruby theme={null}
SearchEngine.configure { |c| c.schema.retention.keep_last = 0 }
```

* Per-collection override:

```ruby theme={null}
class SearchEngine::Book < SearchEngine::Base
  schema_retention keep_last: 2
end
```

After a successful swap, older physicals that match the naming pattern and are not the alias target are ordered by embedded timestamp (desc). Everything beyond the first <code>keep\_last</code> is deleted. The alias target is never deleted.

Typical operational pattern:

1. Run <code>schema:apply</code> (or <code>Schema.apply!</code>) to create a new physical, import, swap alias, then drop old physicals per retention.
2. Avoid manual create/import/alias routines in production unless you also implement a cleanup step; otherwise, old physicals will accumulate.

### Rollback

<code>SearchEngine::Schema.rollback(klass)</code> will swap the alias back to the most recent retained physical (behind the current). If no previous physical exists, it raises an error (e.g., when <code>keep\_last</code> is 0). No collections are deleted during rollback.

### Failed indexation safety

When <code>Schema.apply!</code> is used with the high-level <code>.index\_collection</code> flow, the engine
monitors indexation results. If the result status is not <code>:ok</code> (e.g., <code>:partial</code> or
<code>:failed</code>), the block raises <code>SearchEngine::Errors::IndexationAborted</code>, which prevents the
alias swap. The newly created physical collection is automatically cleaned up by the <code>ensure</code>
block in <code>apply!</code>, preventing orphan accumulation. The same cleanup applies when indexing is
interrupted (e.g., via <code>Ctrl+C</code>).

This ensures the previous (fully-indexed) physical remains active and serving queries even when a fresh reindex encounters errors, without leaving dangling collections behind.

### Orphan pruning

Physical collections from older gem versions or edge cases (e.g., cleanup failure during apply) can accumulate.
<code>Schema.prune\_orphans!</code> scans all collections matching the timestamped naming pattern
(<code>logical\_YYYYMMDD\_HHMMSS\_###</code>) and drops any that are not the target of an alias.

```ruby theme={null}
# Prune orphans for all collections
SearchEngine::Schema.prune_orphans!
# => { dropped: ["books_20250101_120000_001"], kept: ["books_20250315_080000_001"], total_scanned: 12 }

# Scope to a single logical collection
SearchEngine::Schema.prune_orphans!(logical: "books")
```

A rake task is also available (see <a href="/projects/search-engine-for-typesense/v30/cli#commands">CLI</a>):

```bash theme={null}
rails 'search_engine:schema:prune_orphans'         # all collections
rails 'search_engine:schema:prune_orphans[books]'   # scoped
```

See also: <a href="/projects/search-engine-for-typesense/v30/client">Client</a>, <a href="/projects/search-engine-for-typesense/v30/configuration">Configuration</a>, and <a href="/projects/search-engine-for-typesense/v30/compiler">Compiler</a>.

## Troubleshooting

* <strong>Reindex step missing</strong>: Provide a block to <code>apply!</code> or implement <code>klass.reindex\_all\_to(name)</code>.
* <strong>Retention errors</strong>: Ensure <code>keep\_last</code> is set appropriately; rollback requires a previous retained physical.
* <strong>Orphaned physicals accumulating</strong>: Since v30.6.14, failed or interrupted applies auto-delete the new physical. For older orphans, run <code>Schema.prune\_orphans!</code> or the <code>schema:prune\_orphans</code> task.
* <strong>Alias not swapped after apply</strong>: If indexation returned a non-ok status, <code>IndexationAborted</code> prevented the swap and the new physical was cleaned up. Fix import errors and retry.

Backlinks: <a href="https://github.com/lstpsche/search-engine-for-typesense#readme" target="_blank">README</a>, <a href="/projects/search-engine-for-typesense/v30/indexer">Indexer</a>