Haystack 如何应用 metadata 过滤器在检索阶段缩小 Retriever 返回的文档范围【免费下载链接】haystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.项目地址: https://gitcode.com/GitHub_Trending/ha/haystack当你把带 metadata 的文档写入 Document Store 后常见的诉求是查询时只从某个子集里取文档比如只要 2019 年的年报、只要某几家公司的报告。Haystack 的做法是给过滤器定义一套统一语法然后把它传给 Retriever 的filters参数——可以在组件初始化时写死也可以在运行时通过Pipeline.run()动态传入管道会自动把它路由给 Retriever 组件见 metadata-filtering。下面以InMemoryDocumentStore为主路径给出可直接运行的完整流程并用断言验证过滤是否生效。先弄清过滤器的两种字典结构过滤器是一个字典或嵌套字典只允许两种类型定义见 metadata-filteringComparison比较必须包含三个键field文档 meta 字段名如meta.typeoperator、!、、、、、in、not invalue单个值in/not in时为列表。filters {field: meta.type, operator: , value: article}Logic逻辑用于嵌套组合多个条件必须包含operatorNOT、OR、ANDconditionsComparison 或 Logic 字典的列表。一个组合示例type为 article、date落在某时间戳区间内、rating≥ 3且genre属于 economy/politics 或publisher为 nytimesfilters { operator: AND, conditions: [ {field: meta.type, operator: , value: article}, {field: meta.date, operator: , value: 1420066800}, {field: meta.date, operator: , value: 1609455600}, {field: meta.rating, operator: , value: 3}, { operator: OR, conditions: [ { field: meta.genre, operator: in, value: [economy, politics], }, {field: meta.publisher, operator: , value: nytimes}, ], }, ], }注意具体支持哪些算子取决于 Document Store 集成。例如ChromaDocumentStore额外支持contains和not contains但不支持NOT细节要查对应集成的 API reference。最小路径用 FilterRetriever 过滤并断言结果FilterRetriever可以配合任意 Document Store 工作专门按过滤器取文档见 FilterRetriever。下面的示例自带断言运行无报错即说明过滤命中了预期那一篇from haystack import Document from haystack.components.retrievers import FilterRetriever from haystack.document_stores.in_memory import InMemoryDocumentStore docs [ Document(contentPython is a popular programming language, meta{lang: en}), Document( contentpython ist eine beliebte Programmiersprache, meta{lang: de}, ), ] doc_store InMemoryDocumentStore() doc_store.write_documents(docs) retriever FilterRetriever(doc_store) result retriever.run(filters{field: lang, operator: , value: en}) assert documents in result assert len(result[documents]) 1 assert result[documents][0].content Python is a popular programming language这里 filters 在run()里传入也可以放在初始化参数里。两点来自该文档的注意事项FilterRetriever返回所有命中过滤器且不去重、不排序的文档对文档量大的 Store 直接filter_retriever.run({})不传 filters会把全部文档推给下游组件它也不计算相似度分数需要按查询排序时应另配 Ranker 组件。在 pipeline 里给 embedding 检索缩小范围embedding-based 检索走InMemoryEmbeddingRetriever它除query_embedding外接受可选的top_k和filters参数用于narrow down the search space见 InMemoryEmbeddingRetriever。示例使用sentence-transformers-haystack包的 embedder运行前需要pip install sentence-transformers-haystackfrom haystack import Document, Pipeline from haystack.document_stores.in_memory import InMemoryDocumentStore from haystack_integrations.components.embedders.sentence_transformers import ( SentenceTransformersTextEmbedder, SentenceTransformersDocumentEmbedder, ) from haystack.components.retrievers import InMemoryEmbeddingRetriever document_store InMemoryDocumentStore(embedding_similarity_functioncosine) documents [ Document( contentThere are over 7,000 languages spoken around the world today., meta{type: article}, ), Document( contentElephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors., meta{type: article}, ), Document( contentIn certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves., meta{type: report}, ), ] document_embedder SentenceTransformersDocumentEmbedder() documents_with_embeddings document_embedder.run(documents)[documents] document_store.write_documents(documents_with_embeddings) query_pipeline Pipeline() query_pipeline.add_component(text_embedder, SentenceTransformersTextEmbedder()) query_pipeline.add_component( retriever, InMemoryEmbeddingRetriever(document_storedocument_store), ) query_pipeline.connect(text_embedder.embedding, retriever.query_embedding) query How many languages are there? # filters 通过 Pipeline.run 传入管道会把它路由到 retriever 组件 result query_pipeline.run( { text_embedder: {text: query}, retriever: { filters: {field: meta.type, operator: , value: article} }, } ) print(result[retriever][documents])文档中filters经由Pipeline.run()路由给 Retriever 的原始写法来自 metadata-filteringpipeline.run( data{ retriever: { query: Why did the revenue increase?, filters: { operator: AND, conditions: [ {field: meta.years, operator: , value: 2019}, { field: meta.companies, operator: in, value: [BMW, Mercedes], }, ], }, }, }, )判断过滤是否生效检查result[retriever][documents]里的每条 Document其meta.type都应为article。文档未给出固定输出结果上面print的实际内容以你的 embedder 模型计算出的相似度排序为准。BM25侧的InMemoryBM25Retriever同样接受top_k和filters可选参数见 InMemoryBM25Retriever传法一致。不经过 Retriever 直接验证过滤器Document Store 的协议方法filter_documents用来按 filters 返回命中文档见 Document Store适合在接入 pipeline 前先确认数据本身能被过滤。metadata-filtering 给出的QdrantDocumentStore示例filters { operator: AND, conditions: [ {field: meta.type, operator: , value: article}, {field: meta.genre, operator: in, value: [economy, politics]}, ], } results QdrantDocumentStore.filter_documents(filtersfilters)关于field的写法文档示例以meta.前缀为主而FilterRetriever文档中的示例直接写field: lang不带前缀InMemoryDocumentStore的元数据辅助方法见 in_memory/document_store.py 中get_metadata_fields_info等方法的说明接受带或不带meta.前缀的字段名两种写法在 InMemory store 上都可以用。初始化 filters 与运行时 filters 的取舍Retriever 的FilterPolicy决定静态 filtersinit 时设置和动态 filtersrun 时传入如何合并取值见 RetrieversREPLACE默认运行时 filters 完全覆盖初始化时的 filters适合让每次查询动态改变过滤范围MERGE运行时 filters 与初始化 filters 合并进一步收窄结果。filters在 init 和 run 两处都可设置filter_policy在所选 Retriever 的 init 方法中设置。以InMemoryBM25Retriever为例源码 中 init 签名为filters: dict[str, Any] | None None与filter_policy: FilterPolicy FilterPolicy.REPLACErun 时通过apply_filter_policy(self.filter_policy, self.filters, filters)合并行为与文档描述一致。限制与不适用场景InMemoryDocumentStore无外部依赖、适合实验官方文档不推荐用于生产环境换成其他 Store 时算子支持范围以该集成文档为准。过滤器只能缩小候选范围不改变相似度排序FilterRetriever本身不打分需要排序时另接 Ranker。文档中引用的外部 tutorial如 Filtering Documents with Metadata不在本仓库内仓库内的语法依据以上述页面为准。【免费下载链接】haystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.项目地址: https://gitcode.com/GitHub_Trending/ha/haystack创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考