# AWS Marketplace Usage Instructions ## Contents * [Introduction](#introduction) * [Prerequisites](#prerequisites) * [Installation](#installation) * API Module * Socket Module * UI Module * [AWS ECS Installation](#aws-elastic-container-service-ecs-installation) ### Introduction Retable is designed to operate exclusively in UNIX environments. All commands provided here have been tested on Unix-based EC2 instances, regardless of the specific OS chosen. The Retable application comprises three main modules: API, UI, and Socket. Ensure all prerequisites are met before starting the application. Note that the API module is resource-intensive and requires the most allocation. Therefore, allocate a larger portion of your resources to the API module. The sequence for starting the modules should begin with the API, which depends on PostgreSQL and Redis connections. ### Prerequisites #### **Database** A running PostgreSQL Database (minimum version 13) is required. For a small team, at least 1 CPU core and 2 GB RAM should be sufficient. If usage increases or performance issues arise, consider upgrading the server specifications accordingly.. The app uses two databases: one for meta-information and the other for storing table data. #### **Database Creation Options:** * **Manual Database Creation:** * Users can create the required databases manually using the following commands below: * In this case, the database names specified in the[ API environment file](#api-module-environment-file) must match the manually created ones. * During initialization, the API will verify the existence of these databases. If they exist, it will proceed to create the necessary tables, indexes, and procedures automatically. ```sql CREATE DATABASE rtbl_meta; CREATE DATABASE rtbl_data; ``` * **Automatic Database Creation:** * If users do not create the databases manually, the API will create them during initialization based on the names specified in the [environment file](#api-module-environment-file). * All required tables, indexes, and procedures will be generated automatically. For both methods, users can define custom indexes, procedures, and additional configurations as needed. **Database Connection & Security Configuration:** * Specify the database connection details (user, password, database names, port, etc.) in the API environment file. More details can be found in the *API Module Environment File* [section](#api-module-environment-file). * The API module requires a TCP connection to the database server. Ensure that your security group allows traffic between the API and the database on the specified port. #### **Redis (Optional)** A running Redis Server (minimum version 6) is optional and required only for users who want to use collaborative functionalities and scheduled automations. Redis is mainly used for the **socket module** to enable **collaborative functionalities** and for handling **scheduled automations**. If you want these features, you need a running Redis server and must configure it in both the [API](#api-module-environment-file) and [Socket](#socket-module-environment-file) module’s environment files. The API and Socket modules need TCP connection to the Redis server. Configure your security group to allow traffic between the API and Redis on the specified port. #### **Attachments** To store attachments, set up an AWS S3 Bucket. The API module requires the **Bucket Region**, **Access Key**, and **Bucket Name**, which should be configured in the API environment file. Ensure that proper access permissions are set for the bucket. More information is available in the **API Module Environment File** section. The attachment limit per file is 100MB. If you exceed this limit, please [contact us.](https://app.retable.io/form/NADnEVtb7xj1YTvp) #### Creating an S3 Bucket First, create an S3 bucket by following these [instructions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). After creating the attachment bucket, you must create an IAM role to manage your uploads. #### Creating an S3 Policy Retable requires permissions to upload attachments to your S3 bucket. To grant access, you need to create an S3 policy. Follow these steps to create the policy: {% stepper %} {% step %} Go to the **IAM** **Service** in the AWS Console, select **Policies** from the left pane, and click **Create policy**. {% endstep %} {% step %} Select **S3** as the service {% endstep %} {% step %} Under **Actions**, specify the allowed actions. Use the **Filter Actions** search bar to find and select **PutObject**, **GetObject**, and **DeleteObject**. {% endstep %} {% step %} Under Resources, click **Add ARNs**. In the modal that appears, enter your bucket name in the **Resource bucket name** field and select the **Any object name** checkbox. Then, click **Add ARNs** and close the modal. {% endstep %} {% step %} Click **Next** to proceed. Give the policy a descriptive name and, optionally, a description. {% endstep %} {% step %} Click **Create Policy** to finalize the process. {% endstep %} {% endstepper %} #### Creating an IAM S3 User After creating an S3 policy, you need to attach it to an IAM user. Follow these steps to create the IAM user: {% stepper %} {% step %} Go to the **IAM Service** and select **Users** under the **IAM Resources** section. {% endstep %} {% step %} Click **Create User**. On the next page, enter a name for the IAM user (e.g., s3-user) and click **Next**. {% endstep %} {% step %} **Permissions** section, choose **Attach policies directly**. Search for the policy you created earlier, select the checkbox next to it, and click **Next**. {% endstep %} {% step %} Review the details, and if everything is correct, click **Create User**. {% endstep %} {% endstepper %} #### Creating an Access Key To generate an access key for the newly created IAM user, follow these steps: {% stepper %} {% step %} Open the **IAM** Service, click **Users** in the left pane, and select the newly created user from the list. {% endstep %} {% step %} On the user details page, click **Create access key**. {% endstep %} {% step %} Choose **Application running on an AWS compute service** and check the confirmation box. {% endstep %} {% step %} Optionally, add a description tag, then click **Create access key**. {% endstep %} {% step %} On the final page, copy the **Access Key** and **Secret Access Key**. Ensure that you store the Secret Access Key securely, as it will not be shown again. These keys are required for the [API module environment file](#api-module-environment-file). {% endstep %} {% endstepper %} #### **E-Mails** An SMTP server is necessary for sending emails, which is crucial for inviting users to your app. Ensure that the SMTP server details are correctly configured. ### Installation #### [**API Module**](#user-content-fn-1)[^1] 1. **Prepare your database:** Ensure your PostgreSQL database is set up and ready. [More information](#database). 2. **Create the environment file (`app_api_env`):** * Create a file named `app_api_env` in your project directory. * Add the required environment variables in the format `KEY=VALUE`, each on a new line. #### Api Module Environment File: | Key | Description | | --------------------- | -------------------------------------------------------------------------------------------------------------- | | ADMIN\_USER\_EMAIL | Email address of the admin user. This user will be created automatically when Product run | | ADMIN\_USER\_PASSWORD | Password of the admin user | | AWS\_ECS\_REGION | AWS region where ECS task is running (e.g., eu-west-2) | | DB\_NAME | The name of the meta database, which stores metadata | | DB\_DATA\_NAME | The name of the data database, which stores retables table data | | DB\_HOST | Hostname of the Postgres Database Server | | DB\_PORT | Port of the Database Server | | DB\_USER | Username of the Database User | | DB\_PWD | Password of the Database User | | REDIS\_HOST | Hostname of the Redis Server | | REDIS\_PORT | Port number of the Redis Server | | REDIS\_PASSWORD | Password of the Redis Server | | BASE\_URL | Base URL to reach the whole application (e.g., [https://yourdomain.retable.io](https://yourdomain.retable.io/) | | SHARE\_URL | \[will be filled] | | S3\_ACCESS\_KEY | Your AWS Access Key | | S3\_SECRET\_KEY | Your AWS Secret Access Key | | S3\_BUCKET\_REGION | Your AWS S3 Bucket Region (ex: eu-west-1) | | AWS\_BUCKET\_S3 | Your AWS S3 Bucket name | | META\_BASE\_URL | Base Endpoint to reach the API | | EMAIL\_SMTP | The SMTP server address used for sending emails | | EMAIL\_PORT | The port number used to connect to the SMTP server | | EMAIL\_AUTH\_USER | The email used for authentication with the SMTP server | | EMAIL\_SEND\_FROM | The default email address from which emails will be sent | **Example format of the environment file:** ``` ADMIN_USER_EMAIL=admin@retable.io ADMIN_USER_PASSWORD=!.!-RetableRocks-!.! DB_HOST=localhost DB_PORT=5432 DB_USER=postgres DB_PWD=super_secret_pwd REDIS_HOST=localhost REDIS_PORT=6379 REDIS_PASSWORD=super_secret_pwd2 BASE_URL=https:\/\/retable.yourdomain.io SHARE_URL=https:\/\/retable.yourdomaion.io\/v1\/go AWS_BUCKET_UID=bucket_uid AWS_BUCKET_PWD=bucket_pwd AWS_BUCKET_REGION=eu-west-1 AWS_BUCKET_S3=retable_aws_s3_bucket META_BASE_URL=https:\/\/retable-api.yourdomaion.io ``` #### Socket Module 1. **Ensure your Redis is ready**: * [More information](#redis). 2. **Create the environment file (`app_socket_env`)**: * Create a file named `app_socket_env` in your project directory. * Add the required environment variables to this file. Each variable should be on a new line in the format `KEY=VALUE`. #### Socket Module Environment File: | Key | Description | | ------------------------- | ------------------------------------------- | | REDIS\_HOST | Hostname of the Redis | | REDIS\_PORT | Port number of the Redis Server | | REDIS\_PASSWORD | Password of the Redis Server | | PORT | Port of your Socket Http & Websocket Server | | META\_API\_BASE\_ENDPOINT | Base URL of your API Module | **Example format of the environment file:** ``` REDIS_HOST=myredis.host REDIS_PORT=6379 REDIS_PASSWORD=redisserverpass PORT=4500 META_API_BASE_ENDPOINT=https:\/\/retable-api.yourdomaion.io ``` #### UI Module 1. **Ensure your Api and Socket is ready:** * [For Api Module](#api-module) * [For Socket Module](#socket-module) 2. **Create the environment file (`app_ui_env`)**: * Create a file named `app_ui_env` in your project directory. * Add the required environment variables to this file. Each variable should be on a new line in the format `KEY=VALUE`. #### UI Module Environment File: | Key | Description | | ------------------------ | ----------------------------------------- | | REACT\_APP\_API\_PATH | Api Module Base Enpdoint | | REACT\_APP\_BASE\_URL | Base URL that you access your application | | REACT\_APP\_SHARE\_URL | Same with the `REACT_APP_BASE_URL` | | REACT\_APP\_SOCKET\_PATH | Socket Module Base Endpoint | **Example format of the environment file:** ``` REACT_APP_API_PATH=https:\/\/retable-api.yourdomaion.io\/v1 REACT_APP_BASE_URL=https:\/\/yourdomain.io REACT_APP_SHARE_URL=https:\/\/yourdomain.io REACT_APP_SOCKET_PATH=https:\/\/retable-socket.yourdomain.io ``` After creating these environment files, upload them to an S3 bucket. They will be required for the environmentFiles field when defining an ECS Task Definition. ### AWS Elastic Container Service (ECS) Installation Here’s how to get started with ECS Anywhere. You first create a new Amazon ECS cluster with two external worker nodes by installing Amazon ECS agents, System Manager agents, and Docker on virtual machines (VMs). Second, you create an [Amazon S3](https://aws.amazon.com/s3/) bucket to store input and output files for the ECS task and attachment files. Finally, you create[ AWS Identity and Access Management (IAM)](https://aws.amazon.com/iam/) roles with appropriate permissions and task definitions. This is to deploy AWS Marketplace container product on created ECS Anywhere cluster with a external worker node. 1. Subscribe to [Retable product](https://aws.amazon.com/marketplace/pp/prodview-outmdut4qa4gs). This will incur software costs to your AWS account. You can review the product pricing at [Retable – Pricing](https://aws.amazon.com/marketplace/pp/prodview-outmdut4qa4gs) before subscribing. 2. Create IAM role. 3. Create Task definition. 4. Create Amazon ECS cluster and run the task definition on that Amazon ECS cluster. Attention: Pre-install the following tools on remote node: * [AWS CLI](https://aws.amazon.com/cli/) * Curl ### Solution walkthrough This solution walkthrough has the following steps: 1. Prepare the AWS resources. 2. Prepare the infrastructure node. 3. Deploy the container product. #### Step 1: Prepare the AWS resources In this step, you create an ECS Anywhere cluster to host the remote node and the required AWS IAM roles for the remote node and executing tasks. **A. Create an ECS Anywhere cluster using AWS CLI** To create an ECS Anywhere cluster, execute this command in your CLI: ``` aws ecs create-cluster \ --cluster-name retable-ecs-cluster \ --region us-east-1 ``` You should receive output similar to the following: ``` { "cluster": { "clusterArn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:cluster/retable-ecs-cluster", "clusterName": "retable-ecs-cluster", "status": "ACTIVE", "registeredContainerInstancesCount": 0, "runningTasksCount": 0, "pendingTasksCount": 0, "activeServicesCount": 0, "statistics": [], "tags": [], "settings": [ { "name": "containerInsights", "value": "disabled" } ], "capacityProviders": [], "defaultCapacityProviderStrategy": [] } } ``` **B. Create an IAM role** Create a required IAM role “`ecsAnywhereRole`” for remote instances by following the instructions in [IAM permissions for Amazon ECS Anywhere](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere-iam.html#ecs-anywhere-iam-required) or the following steps: {% stepper %} {% step %} Open the IAM console at . In the navigation pane, choose Roles and then choose **Create role**. {% endstep %} {% step %} Choose the **AWS service** role type, and then choose **Systems Manager**. Choose the **Systems Manager** use case and then **Next: Permissions**. {% endstep %} {% step %} permissions policy section, search for and select the **AmazonSSMManagedInstanceCore** and **AmazonEC2ContainerServiceforEC2Role** policies and then choose **Next: Review**. {% endstep %} {% step %} For Add tags (optional), specify any custom tags to associate with the policy and then choose **Next: Review**. {% endstep %} {% step %} For Role name, enter **ECSAnywhereRole** and optionally you can edit the description. Review your role information and then choose **Create role**. {% endstep %} {% step %} Perform a search for the **ECSAnywhereRole** and then select it to view the role details. {% endstep %} {% step %} On the Permissions tab, make sure that **AmazonSSMManagedInstanceCore** policy is added. {% endstep %} {% endstepper %} #### Step 2: Prepare the infrastructure node In this walkthrough, you use a virtual machine running your managed infrastructure. You can use any type of infrastructure that can run any of [the operating systems supported by ECS Anywhere](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere.html). This may include a physical, on-premises server. **A. Register your virtual machines** You can follow the instructions to learn more about how to [register an external instance to an AWS ESC cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere-registration.html). Alternately, to register virtual machine using the AWS Management Console, perform the following instructions: {% stepper %} {% step %} Open the [Amazon ECS console](https://console.aws.amazon.com/ecs/). {% endstep %} {% step %} From the navigation bar, select the **Region** to use. {% endstep %} {% step %} In the navigation pane, choose **Clusters**. {% endstep %} {% step %} On the Clusters page, choose **retable-ecs-cluster** to register your external instance to. {% endstep %} {% step %} Choose the **Infastructure** tab, then choose **Register external instances** on the Container Instances section. {% endstep %} {% step %} On the Step 1: External instances activation details page, complete the following steps: * For Activation key duration (in days), enter the number of days that the activation key remains active. After the number of days you entered passes, the key no longer works when registering an external instance. * For Number of instances, enter the number of external instances that you want to register to your cluster with the activation key. * For Instance role, choose the IAM role to associate with your external instances. We will use the IAM role `ecsAnywhereRole` created in previous step. * Choose **Generate registration command**. You should see a command similar to this: `curl --proto "https" -o "/tmp/ecs-anywhere-install.sh" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh" && bash /tmp/ecs-anywhere-install.sh --region "us-east-1" --cluster "retable-ecs-cluster" --activation-id "" --activation-code ""` * Copy the generated registration command then run this command on your virtual machine that you prepared. {% endstep %} {% endstepper %} **B. Verify the ECS agent is running on virtual machine** Once the installation completed, to verify if the ECS agent is running on virtual machine, run the following command in your virtual machine’s console. ``` docker ps ``` You should receive output similar to the following example output: ``` root@onprem-vm1:/home/awsmpdemo# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f17dd66f70f6 amazon/amazon-ecs-agent:latest "/agent" 2 minutes ago Up 2 minutes (healthy) ecs-agent ``` On the **AWS Management Console**, navigate to the **Infrastructure** tab. Your remote instance should appear on the **Container instances** section. **A. Set up required IAM role for executing ECS task** Create **ecsTaskExecutionRole** by following the instructions at [Creating the task execution IAM role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html#create-task-execution-role). **B. Attach the required permissions** 1. **Metering Usage policy** Attach the **AWSMarketplaceMeteringFullAccess** managed policy to the IAM role **ecsTaskExecutionRole** by running the following command in CLI: ``` aws iam attach-role-policy \ --region us-east-1 \ --role-name ecsTaskExecutionRole \ --policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringFullAccess ``` 2. **S3 Get Policy** Attach the **AmazonS3ReadOnlyAccess** managed policy to the IAM role **ecsTaskExecutionRole** by running the following command in CLI: ``` aws iam attach-role-policy \ --region us-east-1 \ --role-name ecsTaskExecutionRole \ --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess ``` 3. **Metering Usage** ``` aws iam attach-role-policy \ --region us-east-1 \ --role-name ecsTaskExecutionRole \ --policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage ``` **C. Create ECS task definition file to deploy the container product on your ECS Anywhere cluster** See Retable's usage instructions for more information on the command arguments being passed to the container. 1. Create a task definition file. To create an task definition for ECS Anywhere save this as file called AWSMP-Retable-task-execution.json replacing \ with your actual AWS account ID and \ with the name you chose for the environment files. You can follow the instructions [here to find your AWS Account ID](https://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html#FindingYourAccountIdentifiers). The requiresCompatibilities value must be **EXTERNAL** to allow the task to be deployed on external instances. ```json { "family": "retable-app", "containerDefinitions": [ { "name": "api", "image": "709825985650.dkr.ecr.us-east-1.amazonaws.com/retable/retable-api:1.7.2", "portMappings": [ { "containerPort": 4000, "hostPort": 4000, "protocol": "tcp" } ], "essential": true, "environmentFiles": [ { "value": "arn:aws:s3:::bucket_name/folder/api-aws.docker.env", "type": "s3" } ], "healthCheck": { "command": [ "CMD-SHELL", "curl -f http://localhost:4000/ || exit 1" ], "interval": 30, "timeout": 5, "retries": 3 } }, { "name": "ui", "image": "709825985650.dkr.ecr.us-east-1.amazonaws.com/retable/retable-ui:2.0.1", "portMappings": [ { "containerPort": 3000, "hostPort": 3000, "protocol": "tcp" } ], "essential": true, "environmentFiles": [ { "value": "arn:aws:s3:::bucket_name/folder/ui-aws.docker.env", "type": "s3" } ], "healthCheck": { "command": [ "CMD-SHELL", "curl -f http://localhost:3000/ || exit 1" ], "interval": 30, "timeout": 5, "retries": 3 } }, { "name": "socket", "image": "709825985650.dkr.ecr.us-east-1.amazonaws.com/retable/retable-socket:0.9.0", "portMappings": [ { "containerPort": 4500, "hostPort": 4500, "protocol": "tcp" } ], "essential": false, "environmentFiles": [ { "value": "arn:aws:s3:::bucket_name/folder/socket-aws.docker.env", "type": "s3" } ], "dependsOn": [ { "containerName": "api", "condition": "HEALTHY" } ], "healthCheck": { "command": [ "CMD-SHELL", "curl -f http://localhost:4500/ || exit 1" ], "interval": 30, "timeout": 5, "retries": 3 } } ], "executionRoleArn": "arn:aws:iam::aws-account-id:role/ecsTaskExecutionRole", "networkMode": "bridge", "requiresCompatibilities": [ "EXTERNAL" ], "cpu": "2048", "memory": "7673", "runtimePlatform": { "cpuArchitecture": "X86_64", "operatingSystemFamily": "LINUX" } } ``` 2. Register the task definition to your AWS account.\ To do that, enter the following command in your terminal: ``` aws ecs register-task-definition \ --region us-east-1 \ --cli-input-json file://AWSMP-Retable-task-execution.json ``` 3. Run the registered task definition using the following command in your terminal: ``` aws ecs run-task \ --region us-east-1 \ --cluster retable-ecs-cluster \ --launch-type EXTERNAL \ --task-definition retable-app:1 ``` You can monitor the progress in the Amazon ECS console until it runs successfully. [^1]: