Package 'openalexVectorComp'

Description

Creates a configuration object used by the embedding backend adapter.

Usage

backend_config(
  provider = c("hf", "openai", "tei"),
  base_url = NULL,
  model = NULL,
  max_batch_size = NULL,
  timeout = 60,
  retries = 3,
  tei_url = NULL
)

Arguments

Value

A named list with backend configuration.

Description

Uses the configured backend adapter to embed a character vector of texts. For authenticated providers, set OVC_API_TOKEN in the environment. The adapter sends it as a bearer token.

Usage

backend_embed_texts(texts, backend = backend_config())

Arguments

Value

Numeric matrix with one row per text and columns V1..Vd.

Description

Returns normalized backend metadata used by the pipeline.

Usage

backend_info(backend = backend_config())

Arguments

Value

A list with fields provider, model_id, dim, max_batch_size, and raw.

Description

Reads backend configuration from a YAML file and returns a normalized object in the same format as backend_config().

Usage

backend_read(fn = "embed_model.yaml")

Arguments

Details

Supports both the current flat format and legacy nested metadata format.

Value

A backend configuration list compatible with backend_config().

Description

Writes a backend configuration (same shape as returned by backend_config()) to YAML.

Usage

backend_save(backend = backend_config(), fn = "embed_model.yaml")

Arguments

Value

Invisibly returns fn.

Description

Convenience wrapper around backend_config() for a SPECTER2 setup served by a local TEI (text-embeddings-inference) server. The model itself must be prepared and started externally (see inst/scripts/prepare_specter2_merged.py and inst/scripts/start_tei_specter2.sh, and the specter2-setup vignette).

Usage

backend_specter2_tei(
  port = 8080L,
  host = "localhost",
  model = "allenai/specter2_proximity_merged"
)

Arguments

Details

The model argument is metadata only and is recorded in embed_model.yaml and parquet partition paths; TEI itself loads whichever model it was started with.

Value

A backend configuration list compatible with backend_config().

Description

Collect completed OpenAI batch embedding jobs

Usage

batch_collect_openai(
  project_dir,
  backend = backend_config(provider = "openai"),
  label = "corpus",
  verbose = TRUE
)

Arguments

Value

Invisibly returns a list with collection summary.

Description

Inspect OpenAI batch state for a label

Usage

batch_status_openai(project_dir, label = "corpus", refresh_remote = TRUE)

Arguments

Value

A data frame with one row per tracked job.

Description

Preprocesses corpus text, performs preflight request-size checks, splits work into compliant OpenAI batch jobs, submits them, and returns immediately.

Usage

batch_submit_openai(
  project_dir,
  backend = backend_config(provider = "openai"),
  corpus_name = "corpus",
  label = corpus_name,
  batch_size = 5000,
  delete_existing = FALSE,
  text_preprocessor = clean_abstract_for_embedding,
  cleaner_args = list(),
  save_text = TRUE,
  max_requests_per_job = 20000L,
  max_job_bytes = 150 * 1024^2,
  completion_window = "24h",
  verbose = TRUE
)

Arguments

Value

Invisibly returns a list with state path and submission summary.

Description

Sweeps candidate thresholds over scores stored in a Parquet dataset without loading all rows into memory. Uses two passes: first to determine the score range on the labeled subset; second to accumulate confusion counts across a fixed grid of thresholds. Returns the best threshold per the chosen metric.

Usage

calibrate_threshold(
  scores_parquet,
  score_col,
  labels_parquet,
  metric = c("f1", "precision_at_recall"),
  recall_min = 0.8,
  thresholds = NULL,
  n_thresholds = 1001,
  batch_size = 1e+05,
  verbose = TRUE
)

Arguments

Value

List containing the selected threshold (th) and the associated precision, recall, and f1 values.

Examples

## Not run: 
best <- calibrate_threshold(
  scores_parquet = "output/scores/",
  score_col = "ensemble",
  labels_parquet = "output/labels/",
  batch_size = 200000
)
best$th

## End(Not run)

Description

Applies lightweight rule-based cleaning to title/abstract rows and returns embedding-ready text plus a deterministic text_hash.

Usage

clean_abstract_for_embedding(
  df,
  mode = c("lenient", "balanced", "strict"),
  no_abstract_policy = c("keep_title_only", "discard", "conditional"),
  min_chars = NULL,
  min_alpha_ratio = NULL,
  placeholder_patterns = NULL,
  boilerplate_patterns = NULL,
  html_patterns = NULL,
  return_flags = TRUE
)

Arguments

Value

A data frame with at least columns id, text, text_hash. When return_flags = TRUE, also includes text_quality, abstract_raw_present, abstract_kept, discard_reason, and cleaning_mode.

Description

Runs OpenAI batch status/collect for a prepared demo workspace and, when batch embeddings are available, computes direct-vs-batch vector comparison. Comparison artifacts are written to: ⁠project/openai_batch_comparison/label=<label>/⁠.

Usage

demo_finalize_openai_batch(
  demo_dir,
  api_key = NULL,
  label = "corpus_batch",
  refresh_remote = TRUE,
  verbose = TRUE
)

Arguments

Value

Invisibly returns a list containing status/collect summaries, comparison readiness, and output paths.

Description

Computes cosine distance as 1 - similarity_cosine(a, b).

Usage

distance_cosine(a, b)

Arguments

Value

A single numeric distance value, or NA_real_ when undefined.

Description

Reads embeddings from a model-specific dataset and computes cosine distances between all vectors in corpus_label and all vectors in reference_label. A centroid row/column is added to the matrix:

• rows are corpus ids plus "centroid" (corpus centroid),

• columns are reference ids plus "centroid" (reference centroid).

Usage

distance_reference_cosine(
  project_dir,
  embeddings_dir = "model_id=BAAI_bge-small-en-v1.5",
  corpus_label = "corpus",
  reference_label = "reference",
  batch_size = 1e+05,
  max_cells = 5e+07,
  verbose = TRUE
)

Arguments

Details

Embeddings are expected under: ⁠project_dir/embeddings/model_id=<...>/label=<label>/batch=<n>/...⁠

Output file:

• pairwise-cosine.parquet: wide table with first column id (corpus id or "centroid"), reference-id columns, and a final centroid column.

Value

Invisibly the output directory ⁠project_dir/distance_reference_cosine/model_id=<...>/corpus_label=<...>/reference_label=<...>/⁠.

Description

Fits (or loads) a reference-area model from reference_label embeddings and computes squared Mahalanobis distance for rows in corpus_label.

Usage

distance_ridge(
  project_dir,
  reference_label = "reference",
  corpus_label = "corpus",
  fit_path = NULL,
  batch_size = 1e+05,
  regularization = 1e-06,
  verbose = TRUE
)

Arguments

Value

Invisibly the model output directory under ⁠project_dir/distance_ridge/model_id=<...>/corpus_label=<...>/reference_label=<...>/⁠.

Description

Opens two Parquet datasets (prototype margins and ridge scores) as Arrow datasets and performs a lazy inner join on the common key (typically id). The result is an Arrow-dplyr query that is not materialized until you call dplyr::collect() or write it with arrow::write_dataset().

Usage

distances(prototype_distances, ridge_distance)

Arguments

Value

A lazy Arrow dplyr query representing the joined datasets.

Examples

## Not run: 
joined <- distances(
  prototype_distances = "path/to/prototype_distances/",
  ridge_distance      = "path/to/ridge_scores/"
)

# Continue piping lazily and write without loading into memory
joined |>
  dplyr::mutate(ensemble = (margin + relevance_score) / 2) |>
  arrow::write_dataset(path = "path/to/output_scores/", format = "parquet")

# Or collect a small sample for inspection
head(dplyr::collect(joined))

## End(Not run)

Description

Processes a Parquet dataset without loading it fully in memory. Reads Arrow record batches, builds canonical text from title + abstract, calls the configured embedding backend, and writes Parquet batch files.

Usage

embed_corpus(
  project_dir = NULL,
  backend = backend_config(),
  corpus_name = "corpus",
  batch_size = 5000,
  delete_existing = FALSE,
  text_preprocessor = clean_abstract_for_embedding,
  cleaner_args = list(),
  save_text = TRUE,
  label = corpus_name,
  dry_run = FALSE,
  verbose = TRUE
)

Arguments

Value

Invisibly the model-specific embeddings directory under ⁠project_dir/embeddings/model_id=<...>/⁠.

Description

Sends a character vector to the configured backend and returns embeddings as a numeric matrix.

Usage

embed_texts(texts, backend = backend_config())

Arguments

Value

A numeric matrix with one row per input text and one column per embedding dimension.

Description

Fits a reference-area model (centroid + regularized covariance inverse) using rows from reference_label.

Usage

fit_ridge(
  embeddings,
  reference_label = "reference",
  output,
  regularization = 1e-06,
  verbose = TRUE
)

Arguments

Value

Invisibly returns output.

Description

Reads an embeddings Parquet dataset (produced by embed_corpus()) with columns id and V1..Vd, computes a PCA on the embedding matrix, and returns a scatter plot of the first two principal components. Points are colored by labels provided via labels. Rows not found in labels are shown as "other".

Usage

plot_embeddings_pca(
  embeddings,
  labels,
  center = TRUE,
  scale. = FALSE,
  point_size = 2,
  alpha = 0.5
)

Arguments

Value

A ggplot object with points mapped to PC1 vs PC2 and colored by group.

Examples

## Not run: 
p <- plot_embeddings_pca(
  embeddings = "inst/examples/embedings/",
  labels = data.frame(
    id = c("W1", "W2", "W10"),
    label = c("reference", "reference", "corpus")
  )
)
print(p)

## End(Not run)

Description

Computes a 2D UMAP projection of V1..Vd and returns a scatter plot colored by labels membership. Uses cosine distance by default to align with common embedding similarity.

Usage

plot_embeddings_umap(
  embeddings,
  labels,
  n_neighbors = 15,
  min_dist = 0.1,
  metric = "cosine",
  n_epochs = 500,
  seed = 42,
  sample_n = NULL,
  point_size = 2,
  alpha = 0.5
)

Arguments

Value

A ggplot object of UMAP1 vs UMAP2 colored by group.

Description

Uses the same demo structure as run_demo_openalex(), but configures an OpenAI backend and requires an explicit API key argument. The key is set in OVC_API_TOKEN for the duration of the call.

Usage

run_demo_openai(
  api_key,
  demo_dir = file.path(getwd(), "demos", "openai"),
  render = TRUE,
  model = "text-embedding-3-small",
  max_corpus = 100,
  max_reference = 10,
  overwrite = FALSE,
  quarto_file = "openai_demo_analysis.qmd",
  verbose = TRUE
)

Arguments

Value

Invisibly returns a list with project paths and render status.

Description

Sets up a demo workspace under demo_dir, creates a pipeline project under demo_dir/project, copies small corpus and reference fixtures and Quarto template from inst/ovc_demo, and optionally renders the analysis.

Usage

run_demo_openalex(
  demo_dir = file.path(getwd(), "demos", "openalex"),
  render = TRUE,
  backend = NULL,
  max_corpus = 100,
  max_reference = 10,
  overwrite = FALSE,
  quarto_file = "openalex_demo_analysis.qmd",
  verbose = TRUE
)

Arguments

Details

The workspace keeps all generated directories and output artifacts. The Quarto file is created in demo_dir, while embedding pipeline data is stored in demo_dir/project.

Value

Invisibly returns a list with project paths and render status.

Description

Reads a wide reference-cosine distance matrix (as written by distance_reference_cosine()) and converts all numeric distance columns to scores.

Usage

score_reference_cosine(
  distance_parquet,
  output_dir = NULL,
  method = c("linear", "exponential"),
  alpha = 1,
  verbose = TRUE
)

Arguments

Value

Invisibly returns output directory.

Description

Reads a distance dataset with columns id and area_distance, computes relevance_score = exp(-alpha * area_distance), and writes a scored Parquet dataset.

Usage

score_ridge(distance_parquet, output_dir = NULL, alpha = 0.5, verbose = TRUE)

Arguments

Value

Invisibly returns output directory.

Description

Computes cosine similarity for two numeric vectors of equal length. Returns NA_real_ if either vector has zero norm.

Usage

similarity_cosine(a, b)

Arguments

Value

A single numeric similarity value in ⁠[-1, 1]⁠, or NA_real_ when undefined.