Step 06 - Using an external vector store
In the previous step, we implemented a RAG (Retrieval Augmented Generation) pattern in our AI service using an in memory vector store. In this step, we will modify our AI service to use an external vector store.
If you want to see the final result of this step, you can check out the step-06 directory.
Otherwise, let’s get started!
Vector store
There are many options to store the embeddings, like Infinispan, Pinecone, Qdrant, specialized databases (like Chroma), etc. Here, we will use the PostgreSQL pgVector store, a popular relational database. For simplicity, we will run the database with Docker or Podman. If you are not able to use Docker or Podman, you can skip this step and continue to use the in memory vector store.
Configuring the vector store
Add the following dependency to your pom.xml file:
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-pgvector</artifactId>
</dependency>
Now we will be able to use the dev.langchain4j.store.embedding.pgvector.PgVectorEmbeddingStore class to store and retrieve the embeddings.
Before we can use the PgVectorEmbeddingStore class, we need to configure it. In the
src/main/liberty/config directory, create a file named bootstrap.properties with the following content:
vector.store.hostname=localhost
vector.store.port=5432
vector.store.database=postgres
vector.store.username=postgres
vector.store.password=password
vector.store.dimension=384
Let’s look at the configuration:
vector.store.hostname: The hostname where the vector store is running. Since we are running it locally, we specify a value oflocalhost. If you build the AI service into an image and run it in Docker or Podman as well, you will need to specify the hostname or IP address of your local machine.vector.store.port: The port that the vector store is listening on. We will specify this when we run the database container.vector.store.database: The name of the database in the vector store.vector.store.username: The username used to connect to the vector store.vector.store.password: The password used to connect to the vector store.vector.store.dimension: The size of the embeddings that will be stored in the vector store. The value is the size of the vectors generated by the all-minilm-l6-v2 embedding model.
Implementing the vector store
Now let’s create the PgVectorEmbeddingStore bean. Create the
dev.langchain4j.workshop.PgVectorEmbeddingStoreProducer class with the following content:
package dev.langchain4j.workshop;
import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.store.embedding.EmbeddingStore;
import dev.langchain4j.store.embedding.pgvector.PgVectorEmbeddingStore;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.inject.Produces;
import org.eclipse.microprofile.config.inject.ConfigProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* CDI producer for creating and configuring the EmbeddingService instance.
*/
@ApplicationScoped
public class PgVectorEmbeddingStoreProducer {
private static final Logger logger = LoggerFactory.getLogger(PgVectorEmbeddingStoreProducer.class);
/**
* Produces an EmbeddingStore.
*
* @return EmbeddingStore<TextSegment>
* The EmbeddingStore
*/
@Produces
public EmbeddingStore<TextSegment> produceEmbeddingStore(
@ConfigProperty(name = "vector.store.hostname") String hostname,
@ConfigProperty(name = "vector.store.port") int port,
@ConfigProperty(name = "vector.store.database") String database,
@ConfigProperty(name = "vector.store.username") String username,
@ConfigProperty(name = "vector.store.password") String password,
@ConfigProperty(name = "vector.store.dimension") int dimension
) {
logger.info("Creating PgVector EmbeddingStore");
return PgVectorEmbeddingStore.builder()
.host(hostname)
.port(port)
.database(database)
.user(username)
.password(password)
.table("embeddings")
.dimension(dimension)
.build();
}
}
This class simply creates an instance of the PgVectorEmbeddingStore bean using the properties that we defined in the
bootstrap.properties file above.
Note that the produceEmbeddingStore method is annotated with the @Produces annotation. This means that our AI
service will use the PgVectorEmbeddingStore bean created by the produceEmbeddingStore method whenever it needs to
inject a bean EmbeddingStore<TextSegment>.
Also note that the we specify that the bean should use the embeddings table to store and retrieve embeddings.
Updating the ingestor
Now that we have implemented our vector store, we need to update our document ingestor to use it instead of the
in memory vector store. Modify the RAGDocumentIngestor as follows:
@ApplicationScoped
public class RAGDocumentIngestor {
private static final Logger logger = LoggerFactory.getLogger(RAGDocumentIngestor.class);
@Produces
private EmbeddingModel embeddingModel = new AllMiniLmL6V2EmbeddingModel();
@Inject
private EmbeddingStore<TextSegment> embeddingStore;
@Inject
@ConfigProperty(name = "customer-support-agent.rag.docs.dir")
private String ragDocsDir;
@Inject
@ConfigProperty(name = "customer-support-agent.rag.max-segment-size")
private int maxSegmentSize;
@Inject
@ConfigProperty(name = "customer-support-agent.rag.max-overlap-size")
private int maxOverlapSize;
public void ingest(@Observes @Initialized(ApplicationScoped.class) Object pointless) throws URISyntaxException {
long start = System.currentTimeMillis();
// Create the embedding store ingestor
EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder()
.documentSplitter(DocumentSplitters.recursive(maxSegmentSize, maxOverlapSize))
.embeddingModel(embeddingModel)
.embeddingStore(embeddingStore)
.build();
// Load the document(s) from the configured location and ingest them
List<Document> docs = ClassPathDocumentLoader.loadDocuments(ragDocsDir, new TextDocumentParser());
ingestor.ingest(docs);
logger.info("Ingested {} docs in {}ms", docs.size(), System.currentTimeMillis() - start);
}
}
Note that we do not need to modify the configuration for our content retriever. The
src/main/resources/META-INF/microprofile-config.properties file specifies that our doc-retriever bean should use
the default (unqualified) CDI bean for the embedding store. This will now be the the PgVectorEmbeddingStore bean.
Testing the application
Let’s see if everything works as expected.
Podman or Docker
The application requires Podman or Docker to run a PostgreSQL database. So make sure you have one of them installed and running.
First, you need to run the vector store inside Docker or Podman. To start it, run one of the following commands, depending on the environment that you use:
-
Docker:
-
Podman:
Before running the AI service, check that the vector store is running and that the database is clean. Connect to the
postgres database using one of the following commands:
-
Docker:
-
Podman:
This will run the psql command line and connect you to the postgres database. You should see the following output
in your terminal:
You can check that there are currently no tables in the database using the \dt command:
🛠 Troubleshooting – Docker Permission Issue on Linux
If you see this error when running the project:
It means your user doesn’t have access to the Docker socket (/var/run/docker.sock).
✅ Solutions
Quick fix (⚠️ not recommended for production):
This grants full access to all users. Use only in local/dev environments.Recommended fix: Add your user to the docker group:
After adding your user to the Docker group, don’t forget to log out and back in for the changes to take effect.
Now that you have confirmed that the vector store is running, you need to start the AI service. If you stopped the application, restart it with the following command in a different terminal window:
When the application starts, it will ingest the documents into the vector store.
You should still see the following lines in the log that indicate that documents are being ingested. The difference now is that the documents are being ingested into the external vector store rather than the in memory vector store.
[INFO] INFO ai.djl.util.Platform -- Found matching platform from: wsjar:file:/Users/msmiths/.m2/repository/ai/djl/huggingface/tokenizers/0.36.0/tokenizers-0.36.0.jar!/native/lib/tokenizers.properties
[INFO] DEBUG ai.djl.huggingface.tokenizers.jni.LibUtils -- Using cache dir: /Users/msmiths/.djl.ai/tokenizers/0.21.0-0.36.0-cpu-osx-aarch64
[INFO] DEBUG ai.djl.huggingface.tokenizers.jni.LibUtils -- Loading huggingface library from: /Users/msmiths/.djl.ai/tokenizers/0.21.0-0.36.0-cpu-osx-aarch64
[INFO] DEBUG ai.djl.huggingface.tokenizers.jni.LibUtils -- Loading native library: /Users/msmiths/.djl.ai/tokenizers/0.21.0-0.36.0-cpu-osx-aarch64/libtokenizers.dylib
[INFO] INFO dev.langchain4j.liberty.workshop.producers.PgVectorEmbeddingStoreProducer -- Creating PgVector EmbeddingStore
[INFO] DEBUG dev.langchain4j.store.embedding.EmbeddingStoreIngestor -- Starting to ingest 1 documents
[INFO] DEBUG dev.langchain4j.store.embedding.EmbeddingStoreIngestor -- Documents were split into 39 text segments
[INFO] DEBUG dev.langchain4j.store.embedding.EmbeddingStoreIngestor -- Starting to embed 39 text segments
[INFO] DEBUG dev.langchain4j.store.embedding.EmbeddingStoreIngestor -- Finished embedding 39 text segments
[INFO] DEBUG dev.langchain4j.store.embedding.EmbeddingStoreIngestor -- Starting to store 39 text segments into the embedding store
[INFO] DEBUG dev.langchain4j.store.embedding.EmbeddingStoreIngestor -- Finished storing 39 text segments into the embedding store
[INFO] INFO dev.langchain4j.workshop.RAGDocumentIngestor -- Ingested 1 docs in 167ms
To verify this, return to the terminal window that is connected to the vector store database and run the \dt command
again to list the tables in the postgres database:
postgres=# \dt
List of relations
Schema | Name | Type | Owner
--------+------------+-------+----------
public | embeddings | table | postgres
(1 row)
You can see that an embeddings table has been created.
Now verify that the table contains the expected number of embeddings. From the logs above, you should expect to find
39 embeddings (segments) in the embeddings table. Run the following command to verify this:
Let’s test with the chatbot. Open your browser and go to http://localhost:9080. Ask the question to the chatbot and see if it retrieves the relevant segments and builds a cohesive answer:
Conclusion
In this step, modified our RAG implementation to store the embeddings in an external vector store.
In the next step let’s switch to another very popular pattern when using LLMs: Function Calls and Tools.