pgvector

cargo add octopux --features sqlx,pgvector         # sqlx with `postgres`; add graphql for the GraphQL scalars

The vector, halfvec and sparsevec columns of pgvector become the Vector, HalfVector and SparseVector fields of the sqlx models.

The vectors stay on the backend: clients send text and receive rows. The backend embeds the text before a save or a search.

Vector fields

The model sent to the clients leaves the vector out:

#[derive(Serialize, Deserialize, sqlx::FromRow, HttpFindListDelete, SqlxModel)]
#[http_find_list_delete(Id, FindQuery, ListQuery, DeleteQuery, AppState)]
#[sqlx_model(database = "postgres", filter)]
#[octopux_info(path = "article")]
pub struct Article {
    pub id: Id,
    pub title: String,
    pub body: String,
    // no `embedding`: the vector(384) column is never sent
}

The insert and update models hold it, skipped from the payload, and before_save computes it:

use octopux::pgvector::Vector;

#[derive(Serialize, Deserialize, InputObject, HttpCreate, SqlxNewModel)]
#[http_create(SaveQuery, AppState)]
#[sqlx_model(database = "postgres", model = "Article", before_save)]
pub struct NewArticle {
    pub title: String,
    pub body: String,
    #[serde(skip)]
    #[graphql(skip)]
    pub embedding: Option<Vector>,
}

#[async_trait]
impl BeforeSave<AppState> for NewArticle {
    async fn before_save(mut self: Self, state: &AppState) -> Result<Self> {
        self.embedding = state.embedder.embed(vec![format!("{}\n{}", self.title, self.body)]).await?.pop();
        Ok(self)
    }
}

A SqlxFilter field with op = "nearest" orders the list by distance, nearest first; limit keeps the k nearest rows. Skip it from the query string so that only the backend sets it:

#[derive(Default, Deserialize, JsonSchema, ApiComponent, SqlxFilter)]
#[sqlx_filter(database = "postgres")]
pub struct ListQuery {
    pub limit: Option<usize>,
    pub category: Option<String>,
    #[sqlx_filter(column = "embedding", op = "nearest")]
    #[serde(skip)]
    #[schemars(skip)]
    pub near: Option<pgvector::Vector>,
}

The search embeds the text and lists the rows:

pub async fn search_articles(query: SearchQuery, state: &AppState) -> anyhow::Result<Vec<Article>> {
    let near = Some(state.embedder.embed_query(&query.q).await?);
    let list = ListQuery { limit: query.limit, category: query.category, near };
    Ok(Article::list(&list, state).await?)
}

Mount /article/search before the routes of the model, so that search is not taken for an {id}. The other filters still apply.

distance Order HNSW operator class
cosine (default) <=> vector_cosine_ops
l2 <-> vector_l2_ops
inner_product <#>, largest first vector_ip_ops
l1 <+> vector_l1_ops

Index the column with the operator class of the distance (halfvec_*_ops, sparsevec_*_ops for the other types):

CREATE INDEX article_embedding_idx ON article USING hnsw (embedding vector_cosine_ops);

Similar rows

The rows similar to a row reuse its stored vector, nothing is embedded: a HasMany relation reads it, then orders the other rows by distance.

SELECT id, title, 1 - (embedding <=> $2) AS similarity FROM article
WHERE id <> $1 ORDER BY embedding <=> $2 LIMIT $3

GraphQL and OpenAPI

With graphql and openapi, the vector types are scalars and documented schemas. Keep them out of the API anyway: resolvers take text, inputs skip the vector with #[graphql(skip)].

{ searchArticles(q: "ramen in tokyo", limit: 2) { id title } }
mutation { createArticle(input: { title: "Pasta al pesto", body: "Basil, pine nuts..." }) { id } }

CLI and reverse engineering

With --postgres, the CLI offers pgvector::Vector field types: the migration creates the extension, the vector(n) column and its HNSW cosine index. octopux-reverse maps the pgvector columns to these types; hide the fields as above.

Semantic search generated by the CLI

With --sqlx --postgres, generate-model and add-field offer to compute a pgvector::Vector field from the text of the model with fastembed, locally on the CPU:

? `embedding` is a vector: compute it from the text of the model with fastembed, and search the model by meaning? (y/N) y
? Embedding model › [MultilingualE5Small]
? Fields `embedding` is computed from › [title, summary]
Model Dimensions  
MultilingualE5Small 384 multilingual
MultilingualE5Base 768 multilingual, more precise, slower
BGESmallENV15 384 English
BGEBaseENV15 768 English, more precise, slower
AllMiniLML6V2 384 English, fastest

The CLI writes:

  • src/embedder.rs: the fastembed model, shared by all the searches. Your code depends on fastembed, octopux does not: edit it freely.
  • <table>_search.rs: the BeforeSave implementations, GET /<model>/search?q=..., the search<Models> GraphQL query, and embed_missing for the rows saved before.
  • with --bootstrap, the embedder in AppState.

It then offers GET /<model>/{id}/similar, see generate-relation --nearest.

Mount the search first: .configure(book_search::configure).configure(book::configure). The model is downloaded on the first run into .fastembed_cache.

The search always answers the nearest rows, even unrelated ones: filter on similarity. Its scale depends on the model (unrelated rows stay below ~0.8 with E5, ~0.55 with BGE, ~0.3 with AllMiniLML6V2); check a few searches on your data. A vector computed from more text separates better than a title alone.

See the recipes example: a semantic search over 340 recipes.