Connect AnythingLLM to CometAPI for Document Q&A

Last reviewed: 2026-08-25

AnythingLLM gives you a workspace for uploading documents and chatting with them. CometAPI can supply the chat model through AnythingLLM’s Generic OpenAI provider, so you can try a current CometAPI model without writing an SDK adapter. This guide keeps the setup deliberately small: configure the provider, choose a model from the live catalog, run a generation check, and then run a document-grounded question. The two tests are separate; a successful provider response does not by itself prove that retrieval is working.

Direct answer

Install AnythingLLM on the machine where you will run the workspace. The CometAPI integration guide for AnythingLLM lists an installed AnythingLLM instance and an active CometAPI account as prerequisites. In the CometAPI dashboard, create or select an API key and note the displayed base URL. Keep the key in your password manager or the application settings; never paste it into a tutorial, screenshot, issue, or log.

Open Settings → AI Providers → LLM Preferences in AnythingLLM. Select Generic OpenAI, then enter the versioned CometAPI endpoint ending in /v1 in the URL field.

Paste your key into the API Key field without copying it into a shell history or shared notes. In the Model field, use a current model identifier from CometAPI’s model list rather than an identifier copied from an old screenshot. The CometAPI guide says that you can add multiple model names on the same configuration page, which is useful for comparing a fast model with a larger one later.

Set Token Context Window and Max Tokens to values that match the selected model. The source guide intentionally does not prescribe one number for every model; treat the catalog or model documentation as the authority. Save the provider configuration, open a workspace, and send a short test message such as Reply with the word ready. If the workspace receives a normal answer, the provider path is responding.

Now test the document path. Add a small, non-sensitive document with one distinctive fact, wait for AnythingLLM to finish processing it, and ask a question whose answer appears verbatim in that document. Ask a second question whose answer is absent. A useful result should answer the first question from the workspace context and acknowledge that the second cannot be found, rather than inventing a detail. AnythingLLM’s public project description highlights document ingestion, document chat, and source citations; inspect the available source indication in your workspace when judging the result.

Here is a compact configuration and test record. The bracketed values are placeholders, not credentials:

provider: Generic OpenAI
base_url: [VERSIONED_COMETAPI_ENDPOINT]
api_key_present: true
model_id: [CURRENT_MODEL_ID]
token_context_window: [MATCH_MODEL]
max_tokens: [MATCH_MODEL]
workspace_id: [WORKSPACE_ID]
document_test: known_fact_then_absent_fact

A repeatable operator workflow

Use the same sequence whenever you change a model or AnythingLLM version:

  1. Record the provider name, exact base URL, model identifier, and token settings. Record whether a key is present, never the key itself.
  2. Send a short generation-only prompt in a fresh workspace. This isolates authentication, routing, and model selection from retrieval.
  3. Upload a small test document and ask one known-answer question. Save only a citation or source label if the UI provides one; do not copy the document into an operations ticket.
  4. Ask an absent-answer question. Treat a confident answer without evidence as a retrieval-quality failure, even if the provider call succeeded.
  5. If any step fails, capture sanitized fields such as provider, base_url, model_id, status, http_status when available, latency_ms, source_count, and answer_chars. Omit API keys, prompts, document text, and full responses.
  6. Change one setting at a time, rerun the generation check, and then rerun the document check. This leaves a useful trail without exposing user content.

Who this is for

This article is for people who already use, or are evaluating, AnythingLLM as a desktop or self-hosted document-chat workspace and want CometAPI as the language-model backend. It is a good fit for a support engineer validating a new model, a developer preparing a private knowledge-base demo, or a small team that wants to compare model IDs from one provider inside a familiar UI.

It is not a guide to building a custom AnythingLLM plugin, changing the vector database, or configuring an embedding service. Those are separate contracts. The steps here establish the LLM request path first, then give you a simple retrieval test. Keeping those boundaries explicit makes a failure easier to diagnose: an authentication error is not the same problem as a document that was never indexed.

Key takeaways

  • Choose Generic OpenAI because the AnythingLLM documentation describes it as a wrapper for providers that are OpenAI-compatible in both API behavior and inference response.
  • Use the versioned CometAPI endpoint ending in /v1 in the AnythingLLM URL field. The dashboard’s displayed root base URL and the versioned configuration URL serve different purposes in the CometAPI instructions.
  • Select a model ID from the current CometAPI catalog and align both token settings with that model. Do not make a model name permanent just because it appeared in an older example.
  • Prove the setup in two passes: a generation-only message, followed by a known-fact and absent-fact document test.
  • Keep logs operationally useful but content-free. A boolean such as api_key_present: true is safer than recording a key, request header, prompt, or document excerpt.

If you need a preflight for model identifiers, use the site guide to validate the CometAPI model catalog before integration . If your next step is a coded knowledge-base rather than a desktop workspace, compare the semantic FAQ search pattern with CometAPI embeddings and pgvector .

Sources checked

The setup claims in this article are grounded in three public sources:

  1. Use AnythingLLM with CometAPI is the provider-specific source. It names the Generic OpenAI selection, the versioned CometAPI URL, the API Key and model fields, token settings, multiple model names, and the workspace test. The page reports a last-modified date of July 2, 2026.
  2. OpenAI (Generic) LLM is AnythingLLM’s own documentation for the generic wrapper. It cautions that the wrapper is highly configurable and depends on the provider being compatible in both API functionality and inference response.
  3. AnythingLLM on GitHub is the public project repository. Its current README describes document ingestion and chat, source citations, workspaces, vector databases, multiple deployment options, and support for an OpenAI (Generic) provider and CometAPI chat models. It also explains the categories of anonymous telemetry the project reports, which is relevant when deciding what to record locally.

These sources answer different questions: CometAPI supplies the exact integration values, AnythingLLM defines the generic-provider contract, and the project repository establishes what the workspace and document-chat experience is intended to do.

Contract details to verify

URL and provider selection

The CometAPI page displays its API root as the base URL obtained with the key, then instructs you to enter the versioned endpoint ending in /v1 in AnythingLLM’s URL field. Preserve the /v1 suffix in that field. Do not append query parameters or put a key in the URL.

Model and limits

Model identifiers can change independently of the UI. Verify the spelling and availability in the current CometAPI model list immediately before testing. The CometAPI instructions also call out Token Context Window and Max Tokens; keep those values within the selected model’s documented limits. Because the supplied sources do not define universal numeric limits, a tutorial should not hard-code a number and imply that it applies to every model.

Generation versus retrieval

A reply to Reply with the word ready. proves only that AnythingLLM reached a model and received a response. It does not prove that a document was processed, that the workspace selected the intended context, or that the answer is grounded. Run the known-fact and absent-fact checks separately. When the UI exposes a source citation, compare it with the uploaded document and record only a source label or count in operational notes.

Logging and privacy

The project README describes telemetry categories such as provider/model tags, document add or remove events, and chat events, while stating that chat content is not collected by that telemetry. Your own logs are a separate responsibility. A safe minimum is provider, base URL, model ID, status, timing, source count, and answer length. Do not log the API key, authorization header, raw prompt, document text, or full model response. If a support ticket needs a trace, use a short internal trace identifier that cannot be used as a credential.

Failure modes

The connection fails immediately. Recheck that the provider is Generic OpenAI and that the URL matches the documented versioned endpoint ending in /v1. A missing version suffix, an extra path segment, or a copied trailing space can send the request to the wrong route. Retest with a short generation-only message before touching documents.

Authentication is rejected. Confirm that the key is active in the CometAPI dashboard and that it was pasted into the API Key field, not the URL or model field. Replace the key through the dashboard if you suspect exposure. In notes, record only api_key_present: true or false; never include the value.

The model cannot be found. Replace the model field with an identifier currently shown by CometAPI. Avoid guessing from a provider’s marketing name, and remove stale entries when testing multiple models. The CometAPI guide explicitly points readers to its model list and allows multiple model names, so use that list as the source of truth.

The request is cut off or rejected after it starts. Revisit Token Context Window and Max Tokens together. A setting that exceeds the selected model’s contract can produce an unusable test even when the URL and key are correct. Lower or raise one limit according to the model’s current documentation, then rerun the generation check.

The Generic OpenAI wrapper behaves unexpectedly. AnythingLLM labels this provider developer-focused and warns that an incorrect configuration may not function as intended. Reduce the configuration to the documented URL, key, model, and matching limits. Do not add provider-specific headers or query parameters unless a current public contract explicitly requires them.

Generation works but document Q&A is empty or hallucinates. Treat this as a workspace or retrieval investigation. Confirm that the document finished processing, ask a fact that is visibly present, and then ask an absent-fact question. Check the workspace’s source indication if available. Do not “fix” a retrieval issue by increasing model temperature or inventing a model ID; first prove that the document is present and that the selected workspace is the one being tested.

A screenshot or ticket contains sensitive data. Remove the key, authorization material, document excerpts, and full responses before sharing. A sanitized record with field names and status is enough for most routing investigations.

FAQ

Should the URL include /v1?

Yes. The CometAPI AnythingLLM guide distinguishes the root base URL shown with the key from the versioned URL to enter in the Generic OpenAI URL field. Use the versioned endpoint ending in /v1 there.

Can I configure more than one CometAPI model?

Yes. The CometAPI guide says that multiple model names can be added on the same configuration page. Start with one known-current identifier, establish the happy path, and then compare additional models one at a time.

Does this configure embeddings too?

No. These steps configure the chat LLM provider. Document ingestion and retrieval may use a separate embedder and vector-store configuration in AnythingLLM. Keep that configuration separate so you can tell a provider failure from an indexing or retrieval failure.

What counts as a successful document test?

First, a generation-only message should return normally. Next, a known-fact question should produce an answer supported by the uploaded document, and an absent-fact question should not be answered with invented certainty. A source citation or source count, when exposed by the workspace, is useful corroboration but does not replace reading the answer.

Is it safe to put the key in a code sample?

No. Keep it in AnythingLLM’s API Key field and use [REDACTED] in examples, screenshots, logs, and tickets. The URL should contain only the endpoint, never a credential-related query parameter.

Reader next step

Run the two-pass test with a short, non-sensitive document today: generation first, then known-fact and absent-fact questions. If the provider path is not configured yet, use the CometAPI dashboard to select an available model, then return to the URL and token checks above. Keep the resulting operator note content-free so the next model comparison is reproducible without exposing a key or document text.