# Deployment with Existing VPC This release provisions AWS infrastructure for the CREDEBL project using Terraform within an existing VPC. #### Prerequisites 1. Terraform installed. 2. AWS CLI configured with appropriate credentials. 3. Access to the existing VPC ID and subnets. 4. `terraform.tfvars` file with environment-specific values (DEV/QA/PROD). #### Directory Structure ``` terraform-script/ └── aws/ └── modules/ # Terraform modules └── platform/ ├── main.tf # Terraform resources for CREDEBL ├── variables.tf # Input variables ├── backend.tf # Backend configuration for remote state └── terraform.tfvars # Environment-specific values ``` #### Checkout to Release Version Before proceeding with the deployment, make sure you are using the correct tagged release of the Terraform scripts. 1. Clone the repository (if not already done): ```bash git clone https://github.com/credebl/install.git ``` 2. Navigate to the install directory: ```bash cd install ``` 3. Checkout to the release version used for this deployment: ```bash git checkout v2.1.2-alpha.1 ``` > **Note:** This ensures you are deploying the exact version of Terraform scripts tested for CREDEBL infrastructure provisioning.\ > All versions released after **`v2.1.2-alpha.1`** will also be compatible and work with this deployment setup. #### Configuring Environment Variables 1. Open the `terraform.tfvars` file in the `platform` directory: 2. Add or update environment-specific values. For example: ```hcl vpc_id = "vpc-0123456789abcdef" public_subnets = ["subnet-01234abcd", "subnet-05678efgh"] private_subnets = ["subnet-09876ijkl", "subnet-04321mnop"] region = "ap-south-1" project_name = "CREDEBL" environment = "DEV" certificate_arn = "arn:aws:acm:us-west-1:123456789012:certificate/abcd1234-ab12-cd34-ef56-abcdef123456" domain_name = "example.com" ``` > **Note:** Make sure all required variables defined in `variables.tf` are set in `terraform.tfvars`. #### Initialising Terraform 1. Navigate to the Terraform configuration directory: ```bash cd terraform-script/aws/platform ``` 2. Initialise Terraform to download required providers and configure the backend: ```bash terraform init ``` > **Note:** Run `terraform init` whenever providers or modules are updated, or when switching directories. *** #### Planning the Deployment 1. Check the infrastructure changes before applying: ```bash terraform plan ``` * Ensures resources will be created or modified as expected. * Always review the plan output before applying. *** #### Applying the Deployment 1. Provision the infrastructure: ```bash terraform apply ``` 2. Confirm the apply when prompted (`yes`). * Resources will be created in the existing VPC as defined in `terraform.tfvars`. *** #### Infrastructure Provisioned This release will create the required infrastructure on AWS for the CREDEBL platform, using the existing VPC.\ The following services and components will be provisioned: * **ECS Services** * **ECS Task Definitions** * **Application Load Balancer (ALB)** * **Amazon S3 Buckets** * **Security Groups (SG)** * **Elastic File System (EFS)** #### Post-Execution Steps After Terraform execution completes successfully, perform the following manual steps: #### 1. Database Setup You will need **three PostgreSQL databases** for the services: 1. **credo-controller** – Used by the credo controller service. 2. **keycloak** – Used by Keycloak for authentication and identity management. 3. **platform** – Used by the main CREDEBL platform application. You have two options for setting up these databases: * **Option 1:** Create a new PostgreSQL database manually. * **Option 2:** Use an existing database by providing its connection details in the environment files. *** #### 2. Create Environment Files You will need to create **three environment files** for the services, using the environment prefix defined in your `terraform.tfvars` (e.g., `dev`, `qa`, or `prod`). **Example file names:** ``` DEV-credebl.env DEV-credo.env DEV-keycloak.env ``` **File references:** * `DEV-credebl.env` → Refer to:\ \ \ While configuring `DEV-credebl.env` for CREDEBL, the Terraform script provisions services on **ECS Fargate**, so you must update/add the following environment variables: ```bash # Fargate-specific script AFJ_AGENT_SPIN_UP=/agent-provisioning/AFJ/scripts/fargate.sh # AWS and infrastructure references AWS_ACCOUNT_ID= ECS_SECURITY_GROUP_ID= S3_BUCKET_ARN= ECS_SUBNET_ID= FILESYSTEMID= AFJ_VERSION=ghcr.io/credebl/credo-controller:latest INBOUND_TG_ARN= ADMIN_TG_ARN= # Service Connect URLs for ECS services NATS_URL=nats://nats-1-4222.dev-credebl-namespace:4222 REDIS_HOST=redis-sc.dev-credebl-namespace REDIS_PORT=6379 ``` > **Notes:** > > * `NATS_URL` and `REDIS_HOST` point to the ECS Service Connect endpoints. ECS Fargate services will use this for communication. This ensures the **CREDEBL ECS service** can start properly with the correct **network, security, and service connectivity configuration**. * `DEV-credo.env` → Refer to:\ * `DEV-keycloak.env` → Use the following template: ```bash KEYCLOAK_ADMIN=admin KEYCLOAK_ADMIN_PASSWORD=admin KC_HTTP_ENABLED=true KC_DB=postgres KC_DB_URL=jdbc:postgresql://HOST:PORT/Database KC_DB_USERNAME=DB_USERNAME KC_DB_PASSWORD=DB_PASSWORD KC_DB_URL_PORT=PORT PROXY_ADDRESS_FORWARDING=true KC_HOSTNAME_ADMIN_URL=https://KEYCLOAK_DOMAIN KC_HOSTNAME_URL=https://KEYCLOAK_DOMAIN KC_PROXY=edge KC_HOSTNAME_STRICT=false KC_LOG=console KC_HOSTNAME_STRICT_HTTPS=false KC_HTTPS_ENABLED=true ./kcadm.sh config credentials --server http://0.0.0.0:8080 --realm master --user admin ./kcadm.sh update realms/master -s sslRequired=NONE ``` > **Note:** Replace placeholders like `HOST`, `PORT`, `Database`, and `KEYCLOAK_DOMAIN` with actual values. *** #### 3. Upload Environment Files to S3 After creating the three `.env` files, upload them to the S3 bucket created by Terraform.\ The bucket name is: ``` credebl-dev-env-file-bucket ``` > **Note:** Replace `dev` with your environment name if different (e.g., `qa`, `prod`). #### 4. Bastion and EFS Configuration 1. **Create a Bastion Server** * You can either create a new **Bastion EC2 instance** in the same VPC or use an existing one. * Ensure the Bastion host has access to both **NATS EFS** and **CREDO EFS** created by Terraform. 2. **Mount EFS File Systems on Bastion** * Mount both EFS volumes: * **NATS EFS** * **CREDO EFS** * Example command to mount EFS: ```bash sudo mkdir /mnt/nats-efs /mnt/credo-efs sudo mount -t efs fs-xxxxxxx:/ /mnt/nats-efs sudo mount -t efs fs-yyyyyyy:/ /mnt/credo-efs ``` 3. **Configure the EFS Contents** * **In CREDO EFS:** * Create a folder named `token` inside the mounted directory: ```bash mkdir -p /mnt/credo-efs/token ``` * **In NATS EFS:** * Add the `nats.config` file in the root path of EFS.\ Reference:\ * It’s recommended to add authentication to your NATS setup. * To create your Nkeys, you can refer NATS tool nk as per their [official documentation](https://docs.nats.io/using-nats/nats-tools/nk) * Create a `seed` folder and add the `credebl-master-table.json` file: ```bash mkdir -p /mnt/nats-efs/seed ``` Reference for JSON file:\ \ To fill data for your setup, refer to the [credebl documentation](https://docs.credebl.id/docs/contribute/setup/service-setup#installations).\\

If you are using NKeys for authentication:
Add one key from the generated NKey pair to your .env file and add the other key to the nats-server.conf file.

Example:

# In credebl.env
       USER_NKEY_SEED=SUAM6N6D6IODRE2SGUFUGDUKRI7WRLMW2VU4OR2JAGCDGWVURMHBK2ZMGM
       
# In nats-server.conf
       authorization {
         users = [
           { nkey: "UCQDGGRDXSCC777ZTCO2VF6LOK5OC3VSBRMK4ZUTYLKPSI7YKD67SGG6" }
         ]
       }
#### 5. Start Services After completing the Bastion and EFS setup, the services can now be started in the correct order. **Step 5.1: Start Keycloak Service** 1. Start the Keycloak service first. 2. Configure Keycloak following the guide:\ [Keycloak Setup Documentation](https://docs.credebl.id/docs/contribute/setup/service-setup#keycloak) 3. After configuring Keycloak, **update the `.env` file** with the Keycloak credentials.\ Example in `credebl.env`: ```bash KEYCLOAK_DOMAIN=https://KEYCLOAK_DOMAIN/ KEYCLOAK_ADMIN_URL=https://KEYCLOAK_DOMAIN KEYCLOAK_MASTER_REALM=master KEYCLOAK_MANAGEMENT_CLIENT_ID=adminClient KEYCLOAK_MANAGEMENT_CLIENT_SECRET= KEYCLOAK_REALM=credebl-platform ``` *** **Step 5.2: Start Seed Service** 1. Start the **Seed Service**. This service is used **only once** to populate initial data in the database. 2. Monitor the logs. You should see a message like: ``` The seed command has been executed ``` 3. Once the seed command completes, **stop the seed service**, as it is not required to run continuously. > **Note:** The Seed Service writes initial data to your database (CREDEBL tables) and will not be needed after this one-time execution. **Step 5.3: Start NATS, Redis, and Remaining Services** 1. Start the **NATS service** first. * Ensure NATS is correctly configured with the `nats.config` and NKeys authentication is enabled. 2. Once NATS is running, start the **Redis service**. 3. After Redis is running, you can start **all other services** except the following, which should **not be started yet**: * `agent-service` * `agent-provisioning-service` > **Note:** `agent-service` and `agent-provisioning-service` will be started later, after the main platform services are confirmed to be healthy. This ensures the **correct service startup order**: 1. Keycloak → 2. Seed Service → 3. NATS → 4. Redis → 5. All remaining services (excluding agent-related services). **Step 5.4: Start Agent-Provisioning and Agent Services** 1. Start the **agent-provisioning service** first. 2. Once the agent-provisioning service has started successfully, start the **agent service**. * This will executes a script that provisions the necessary agent resources in the **CREDO Controller cluster.** (creates credo-controller service) #### 6. Verification & Validation After all services have been started successfully, perform the following checks to verify that the infrastructure and services are healthy and functioning correctly. 1. Check ECS Services Health: Verify that all ECS services are in the **Running** state. 2. Check Application Load Balancer (ALB): * The **Target Groups** are healthy. * Each registered target shows **“Healthy”** status. ✅ **If all the above checks pass**, your CREDEBL infrastructure is successfully provisioned and all services are up and running on AWS. #### 7. Troubleshooting & Firewall Configuration If your **existing VPC** has a **firewall** or **network restrictions** that block outbound internet access, some resources may fail to pull container images or communicate between services. *** **7.1 Whitelist Required Domains** If services cannot connect to the internet to download images or access container registries, whitelist the following domains in your **firewall policy** or **network access rules**: | Purpose | Domains to Whitelist | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | | **AWS ECR (Public)** |

\*.ecr.aws dkr.ecr.us-east-1.amazonaws.com .cloudfront.net

.s3.amazonaws.com

| | **GitHub Container Registry (GHCR)** | `ghcr.io` | | **Redis Image Repository** | `registry-1.docker.io` | | **NATS Image Repository** | `registry-1.docker.io` | | **Keycloak Image Repository** | `quay.io` | > **Hint:** If your environment uses a proxy, ensure these domains are accessible via the proxy for ECS to successfully pull container images. *** **7.2 Fixing Service Connectivity Issues** If ECS services are unable to connect to **NATS**, **Redis**, or if the **ALB health checks** are repeatedly showing as unhealthy, review and update your **Security Group (SG)** configurations. 1. Identify the Security Groups created by Terraform: * `credo-sg` * `application-sg` -(API-gateway SG) * `studio-sg` * `keycloak-sg` 2. For each service, add an **Inbound Rule**: * **Port:** Service-specific port (for example, 8080 for keycloak service, 3000 for frontend, etc.) * **Source:** VPC CIDR range (e.g., `10.0.0.0/16`) This allows internal traffic between ECS services and ALB within the same VPC. *** **7.3 Fixing NATS and Redis Service Connect Issues** If ECS **Service Connect** is not functioning properly or other services cannot communicate with **NATS** or **Redis**, add explicit inbound rules to their Security Groups: | Service | Port | Source | Purpose | | --------- | ------ | ------------------------------ | --------------------------------------- | | **NATS** | `4222` | VPC CIDR (e.g., `10.0.0.0/16`) | Allows ECS services to connect to NATS | | **Redis** | `6379` | VPC CIDR (e.g., `10.0.0.0/16`) | Allows ECS services to connect to Redis | *** ✅ **After updating the rules or firewall**, re-run your ECS services and verify: * All container images are successfully pulledyouryouryouryour. * NATS, Redis, and ALB health checks are now **Healthy**. * Internal service-to-service communication works without timeouts or connection errors.