📝 Update readme with new config file

README.md

@@ -1,14 +1,52 @@
 # generic-RAG-demo
 
-A Sogeti Nederland generic RAG demo
+A generic Retrieval Augmented Generation (RAG) demo from Sogeti Netherlands built in Python. This project demonstrates how to integrate and run different backends, from cloud providers to local models, to parse and process your PDFs, web data, or other text sources.
+
+## Table of Contents
+
+- [generic-RAG-demo](#generic-rag-demo)
+  - [Table of Contents](#table-of-contents)
+  - [Features](#features)
+  - [Getting started](#getting-started)
+    - [Installation of system dependencies](#installation-of-system-dependencies)
+      - [Unstructered PDF loader (optional)](#unstructered-pdf-loader-optional)
+      - [Local LLM (optional)](#local-llm-optional)
+    - [Running generic RAG demo](#running-generic-rag-demo)
+    - [config.yaml file](#configyaml-file)
+    - [Chainlit starters](#chainlit-starters)
+  - [Dev details](#dev-details)
+    - [Linting](#linting)
+
+## Features
+
+- **Multi-backend Support:** Easily switch between cloud-based and local LLMs.
+- **Flexible Data Input:** Supports both PDFs and web data ingestion.
+- **Configurable Workflows:** Customize settings via a central `config.yaml` file.
 
 ## Getting started
 
 ### Installation of system dependencies
 
+This project leverages a modern packaging method defined in `pyproject.toml`. After cloning the repository, you can install the project along with its dependencies. You have two options:
+1. Using uv
+
+If you're using uv, simply run:
+```bash
+uv install
+```
+
+2. Using a Python Virtual Environment
+
+Alternatively, set up a virtual environment and install the project:
+```bash
+python -m venv .venv        # Create a new virtual environment named ".venv"
+source .venv/bin/activate   # Activate the virtual environment (use ".venv\Scripts\activate" on Windows)
+pip install .              # Install the project and its dependencies
+```
+
 #### Unstructered PDF loader (optional)
 
-If you would like to run the application using the unstructered PDF loader (`--unstructured-pdf` flag) you need to install two system dependencies.
+If you would like to run the application using the unstructered PDF loader (`pdf.unstructured` setting) you need to install two system dependencies.
 
 - [poppler-utils](https://launchpad.net/ubuntu/jammy/amd64/poppler-utils)
 - [tesseract-ocr](https://github.com/tesseract-ocr/tesseract?tab=readme-ov-file#installing-tesseract)
@@ -21,18 +59,19 @@ sudo apt install poppler-utils tesseract-ocr
 
 #### Local LLM (optional)
 
-If you would like to run the application using a local LLM backend (`-b local` flag), you need to install Ollama.
+If you would like to run the application using a local LLM backend (`local` settings), you need to install Ollama.
 
 ```bash
 curl -fsSL https://ollama.com/install.sh | sh  # install Ollama
 ollama pull llama3.1:8b  # fetch and download as model
 ```
 
-Include the downloaded model in the `.env` file:
+Include the downloaded model in the `config.yaml` file:
 
-```text
+```yaml
-LOCAL_CHAT_MODEL="llama3.1:8b"
+local:
-LOCAL_EMB_MODEL="llama3.1:8b"
+    chat_model: "llama3.1:8b"
+    emb_model: "llama3.1:8b"
 ```
 
 >For more information on installing Ollama, please refer to the Langchain Local LLM documentation, specifically the [Quickstart section](https://python.langchain.com/docs/how_to/local_llms/#quickstart).
@@ -52,14 +91,19 @@ python generic_rag/app.py -p data  # will work and parsers all pdf files in ./da
 python generic_rag/app.py --help  # will work and prints command line options
 ```
 
-Please configure your `.env` file with your cloud provider (backend) of choice and set the `--backend` flag accordingly.
+Please configure your `config.yaml` file with your cloud provider (backend) of choice. See the `config.example.yaml` file as a starting point that holds all possible options.
 
-### .env file
+### config.yaml file
 
-A .env file needs to be populated to configure API end-points or local back-ends using environment variables.
+A config.yaml file is required to specify your API endpoints, local backends, and environment variables. Use the provided config.yaml.example as a starting point. Update the file according to your backend settings and project requirements.
-Currently all required environment variables are defined in code at [backend/models.py](generic_rag/backend/models.py)
+
-with the exception of the API key variables itself.
+Key configuration points include:
-More information about configuring API endpoints for langchain can be found at the following locations.
+- Chat Backend: Choose among azure, openai, google_vertex, aws, or local.
+- Embedding Backend: Configure the embedding models similarly.
+- Data Processing Settings: Define PDF and web data sources, chunk sizes, and overlap.
+- Vector Database: Customize the path and reset behavior.
+
+For more information on configuring Langchain endpoints and models, please see:
 
 - [langchain cloud chat model doc](https://python.langchain.com/docs/integrations/chat/)
 - [langchain local chat model doc](https://python.langchain.com/docs/how_to/local_llms/)
@@ -67,27 +111,6 @@ More information about configuring API endpoints for langchain can be found at t
 
 > for local models we currently use Ollama
 
-An `.env` example is as followed.
-
-```text
-# only one backend (azure, google, local, etc) is required. Please addjust the --backend flag accordingly
-
-AZURE_OPENAI_API_KEY="<secret_key>"
-AZURE_LLM_ENDPOINT="https://<project_hub>.openai.azure.com"
-AZURE_LLM_DEPLOYMENT_NAME="gpt-4"
-AZURE_LLM_API_VERSION="2025-01-01-preview"
-AZURE_EMB_ENDPOINT="https://<project_hub>.openai.azure.com"
-AZURE_EMB_DEPLOYMENT_NAME="text-embedding-3-large"
-AZURE_EMB_API_VERSION="2023-05-15"
-
-LOCAL_CHAT_MODEL="llama3.1:8b"
-LOCAL_EMB_MODEL="llama3.1:8b"
-
-# google vertex AI does not use API keys but a seperate authentication method
-GOOGLE_GENAI_CHAT_MODEL="gemini-2.0-flash"
-GOOGLE_GENAI_EMB_MODEL="models/text-embedding-004"
-```
-
 ### Chainlit starters
 
 Chainlit suggestions (starters) can be set with the `CHAINLIT_STARTERS` environment variable.
@@ -102,4 +125,4 @@ CHAINLIT_STARTERS=[{"label":"Label 1","message":"Message one."},{"label":"Label
 
 ### Linting
 
-Currently [Ruff](https://github.com/astral-sh/ruff) is used as Python linter. It is included in the [pyproject.toml](pyproject.toml) as `dev` dependency if your IDE needs that. However, for VS Code a [Ruff extension](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff) excists.
+Currently [Ruff](https://github.com/astral-sh/ruff) is used as Python linter. It is included in the [pyproject.toml](pyproject.toml) as `dev` dependency if your IDE needs that. However, for VS Code a [Ruff extension](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff) exists.