A class that extends the BaseDocumentLoader and implements the DocumentLoader interface. It represents a document loader that loads documents from an Apify dataset.

Example

const loader = new ApifyDatasetLoader("your-dataset-id", {
datasetMappingFunction: (item) =>
new Document({
pageContent: item.text || "",
metadata: { source: item.url },
}),
clientOptions: {
token: "your-apify-token",
},
});

const docs = await loader.load();

const chain = new RetrievalQAChain();
const res = await chain.invoke({ query: "What is LangChain?" });

console.log(res.text);
console.log(res.sourceDocuments.map((d) => d.metadata.source));

Type Parameters

  • Metadata extends Record<string, any>

Hierarchy

Implements

Constructors

Properties

apifyClient: ApifyClient
caller: AsyncCaller
datasetId: string
datasetMappingFunction: ApifyDatasetMappingFunction<Metadata>

Methods

  • Retrieves the dataset items from the Apify platform and applies the datasetMappingFunction to each item to create an array of Document instances.

    Returns Promise<Document<Metadata>[]>

    An array of Document instances.

  • Loads the documents and splits them using a specified text splitter.

    Parameters

    Returns Promise<Document<Record<string, any>>[]>

    A Promise that resolves with an array of Document instances, each split according to the provided TextSplitter.

  • Create an ApifyDatasetLoader by calling an Actor on the Apify platform and waiting for its results to be ready.

    Type Parameters

    • Metadata extends Record<string, any>

    Parameters

    • actorId: string

      The ID or name of the Actor on the Apify platform.

    • input: Record<string | number, unknown>

      The input object of the Actor that you're trying to run.

    • config: {
          datasetMappingFunction: ApifyDatasetMappingFunction<Metadata>;
          callOptions?: ActorCallOptions;
          clientOptions?: ApifyClientOptions;
      }
      • datasetMappingFunction: ApifyDatasetMappingFunction<Metadata>
      • Optional callOptions?: ActorCallOptions
      • Optional clientOptions?: ApifyClientOptions

    Returns Promise<ApifyDatasetLoader<Metadata>>

    An instance of ApifyDatasetLoader with the results from the Actor run.

  • Create an ApifyDatasetLoader by calling a saved Actor task on the Apify platform and waiting for its results to be ready.

    Type Parameters

    • Metadata extends Record<string, any>

    Parameters

    • taskId: string

      The ID or name of the task on the Apify platform.

    • input: Record<string | number, unknown>

      The input object of the task that you're trying to run. Overrides the task's saved input.

    • config: {
          datasetMappingFunction: ApifyDatasetMappingFunction<Metadata>;
          callOptions?: TaskCallOptions;
          clientOptions?: ApifyClientOptions;
      }
      • datasetMappingFunction: ApifyDatasetMappingFunction<Metadata>
      • Optional callOptions?: TaskCallOptions
      • Optional clientOptions?: ApifyClientOptions

    Returns Promise<ApifyDatasetLoader<Metadata>>

    An instance of ApifyDatasetLoader with the results from the task's run.

Generated using TypeDoc