🔜AWS Marketplace Usage Instructions

This page serves as a comprehensive guide for setting up and installing Retable, a Unix-based application for data management. It outlines the system prerequisites, and installation steps for modules.

Contents

• Introduction

• Prerequisites

• Installation

  • API Module

  • Socket Module

  • UI Module

• AWS ECS Installation

  • Fargate

  • EC2

Introduction

Retable is designed to operate exclusively in UNIX environments. All commands provided here have been tested on Unix-based EC2 instances, regardless of the specific OS chosen.

The Retable application comprises three main modules: API, UI, and Socket. Ensure all prerequisites are met before starting the application. Note that the API module is resource-intensive and requires the most allocation. Therefore, allocate a larger portion of your resources to the API module. The sequence for starting the modules should begin with the API, which depends on PostgreSQL and Redis connections.

Prerequisites

Docker

Ensure Docker is installed on your machine. You can download it from the Docker website.

Database

A running PostgreSQL Database (minimum version 13) is required. Based on the Retable App's API module usage, a minimum of 8 GB RAM and 2 CPU cores is recommended. You may need to increase the server specifications if necessary. The app uses two databases: one for meta-information and the other for storing table data.

Example database creation commands:

sqlCopy codeCREATE DATABASE rtbl_meta;
CREATE DATABASE rtbl_data;

When the API module starts, it creates an admin user if one does not already exist. Set the admin email and password in the API environment file. This admin information is used to access the app and invite other users.

Specify database connection information (user, password, database names, port, etc.) in the environment file. More details are available in the API Module Environment File Section.

The API module requires TCP connection to the database server. Configure your security group to allow traffic between the API and the Database on the specified port.

Redis

A running Redis Server (minimum version 6) is needed for collaborative functionalities and scheduled automations.

Specify the connection information in the environment file. More details are available in the API Module Environment File Section.

The API module needs TCP connection to the Redis server. Configure your security group to allow traffic between the API and Redis on the specified port.

Attachments

To store attachments, set up an AWS S3 Bucket. The API module requires the Bucket Region, Access Key, and Bucket Name, which should be configured in the API environment file. Ensure proper access permissions are set for the bucket. More information is available in the API Module Environment File Section.

The attachment limit per file is 100MB. If you exceed this limit, please contact us.

E-Mails

An SMTP server is necessary for sending emails, which is crucial for inviting users to your app. Ensure that the SMTP server details are correctly configured.

Installation

1. Prepare your database: Ensure your PostgreSQL database is set up and ready. More information.

2. Create the environment file (app_api_env):

   • Create a file named app_api_env in your project directory.

   • Add the required environment variables in the format KEY=VALUE, each on a new line.

3. Obtain the Docker image:

   • Access the Amazon ECR repository XXX/retable/rtbl-api.

   • Authenticate Docker to your ECR registry following AWS ECR documentation.

4. Run the Docker container:

   • Navigate to the directory where your app_api_env file is located.

   • Replace <image_tag> with the desired Docker image tag.

   • Execute the command:

   docker run --env-file ./app_api_env -p 4000:4000 XXX/retable/rtbl-api:<image_tag>

   • This command sets up and runs a containerized application with the necessary environment variables and port configurations. Make sure to replace <image_tag> with the actual tag of the image you want to deploy.

Api Module Environment File:

Key	Description
ADMIN_USER_EMAIL	Email address of the admin user. This user will be created automatically when Product run
ADMIN_USER_PASSWORD	Password of the admin user
DB_HOST	Hostname of the Postgres Database Server
DB_PORT	Port nubmer of the Database Server
DB_USER	Username of the Database User
DB_PWD	Password of the Database User
DB_SSL	[will be filled]
REDIS_HOST	Hostname of the Redis Server
REDIS_PORT	Port number of the Redis Server
REDIS_PASSWORD	Password of the Redis Server
DB_SSL_CERT_PATH	[will be filled]
BASE_URL	Base URL to reach the whole application (ex: https://yourdomain.retable.io
SHARE_URL	[will be filled]
AWS_BUCKET_UID	Your AWS Access Key (change this naming)
AWS_BUCKET_PWD	Your AWS Secret Access Key (change this naming)
AWS_BUCKET_REGION	Your AWS S3 Bucket Region (ex: eu-west-1)
AWS_BUCKET_S3	Your AWS S3 Bucket name (change this naming)
META_BASE_URL	Base Endpoint to reach the API

Example format of the environment file:

ADMIN_USER_EMAIL=[e-mail]
ADMIN_USER_PASSWORD=!.!-RetableRocks-!.!

DB_HOST=localhost
DB_PORT=5432
DB_USER=postgres
DB_PWD=super_secret_pwd
DB_SSL=true
DB_SSL_CERT_PATH=eu-west-2-bundle.pem

REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=super_secret_pwd2

BASE_URL=https:\/\/retable.yourdomain.io
SHARE_URL=https:\/\/retable.yourdomaion.io\/v1\/go

AWS_BUCKET_UID=bucket_uid
AWS_BUCKET_PWD=bucket_pwd
AWS_BUCKET_REGION=eu-west-1
AWS_BUCKET_S3=retable_aws_s3_bucket

META_BASE_URL=https:\/\/retable-api.yourdomaion.io

• Pull Api module docker image with this command:

docker pull XXX/retable/rtbl-api:<image_tag>

• To run the the Image

docker run --env-file ./app_env -p 4000:4000 XXX/retable/rtbl-api:<image_tag>

Socket Module

1. Ensure your Redis is ready:

   • More information.

2. Create the environment file (app_socket_env):

   • Create a file named app_socket_env in your project directory.

   • Add the required environment variables to this file. Each variable should be on a new line in the format KEY=VALUE.

3. Obtain the Docker image:

   • Make sure you have access to the Amazon ECR repository XXX/retable/rtbl-socket.

   • Authenticate Docker to your ECR registry. This typically involves running a command to get a temporary login token, which you can find in the AWS ECR documentation.

4. Run the Docker container:

   • Open a terminal and navigate to the directory where your app_socket_env file is located.

   • Replace <image_tag> with the desired tag of the Docker image you want to run.

   • Execute the command:

   docker run --env-file ./app_api_env -p 4000:4000 XXX/retable/rtbl-socket:<image_tag>

   • This command sets up and runs a containerized application with the necessary environment variables and port configurations. Make sure to replace <image_tag> with the actual tag of the image you want to deploy.

Socket Module Environment File:

Key	Description
REDIS_HOST	Hostname of the Redis
REDIS_PORT	Port number of the Redis Server
REDIS_PASSWORD	Password of the Redis Server
PORT	Port of your Socket Http & Websocket Server
META_API_BASE_ENDPOINT	Base URL of your Api Module

Example format of the environment file:

REDIS_HOST=myredis.host
REDIS_PORT=6379
REDIS_PASSWORD=redisserverpass

PORT=4500

META_API_BASE_ENDPOINT=https:\/\/retable-api.yourdomaion.io

UI

1. Ensure your Api and Socket is ready:

   • For Api Module

   • For Socket Module

2. Create the environment file (app_ui_env):

   • Create a file named app_ui_env in your project directory.

   • Add the required environment variables to this file. Each variable should be on a new line in the format KEY=VALUE.

3. Obtain the Docker image:

   • Access the Amazon ECR repository XXX/retable/rtbl-ui.

   • Authenticate Docker to your ECR registry following AWS ECR documentation.

4. Run the Docker container:

   • Navigate to the directory where your app_ui_env file is located.

   • Replace <image_tag> with the desired Docker image tag.

   • Execute the command:

   docker run --env-file ./app_api_env -p 4000:4000 XXX/retable/rtbl-ui:<image_tag>

   • This command sets up and runs a containerized application with the necessary environment variables and port configurations. Make sure to replace <image_tag> with the actual tag of the image you want to deploy.

UI Module Environment File:

Key	Description
REACT_APP_API_PATH	Api Module Base Enpdoint
REACT_APP_BASE_URL	Base URL that you access your application
REACT_APP_SHARE_URL	Same with the REACT_APP_BASE_URL
REACT_APP_SOCKET_PATH	Socket Module Base Endpoint

Example format of the environment file:

REACT_APP_API_PATH=https:\/\/retable-api.yourdomaion.io\/v1
REACT_APP_BASE_URL=https:\/\/yourdomain.io
REACT_APP_SHARE_URL=https:\/\/yourdomain.io
REACT_APP_SOCKET_PATH=https:\/\/retable-socket.yourdomain.io

AWS Elastic Container Service (ECS) Installation

Here’s how to get started with ECS Anywhere. You first create a new Amazon ECS cluster with two external worker nodes by installing Amazon ECS agents, System Manager agents, and Docker on virtual machines (VMs). Second, you create an Amazon S3 bucket to store input and output files for the ECS task and attachment files. Finally, you create AWS Identity and Access Management (IAM) roles with appropriate permissions and task definitions. This is to deploy AWS Marketplace container product on created ECS Anywhere cluster with a external worker node. The following diagram shows the steps to deploy this solution:

1. Subscribe to Retable product. This will incur software costs to your AWS account. You can review the product pricing at Retable – Pricing before subscribing.

2. Create IAM role.

3. Create S3 buckets.

4. Create Task definition.

5. Create Amazon ECS cluster and run the task definition on that Amazon ECS cluster.

Attention: Pre-install the following tools on remote node:

• AWS CLI

• Curl

Solution walkthrough

This solution walkthrough has the following steps:

1. Prepare the AWS resources.

2. Prepare the infrastructure nodes.

3. Deploy the container product.

Step 1: Prepare the AWS resources

In this step, you create an ECS Anywhere cluster to host the remote node and the required AWS IAM roles for the remote node and executing tasks.

A. Create an ECS Anywhere cluster using AWS CLI

To create an ECS Anywhere cluster, go to AWS Cloud9 integrated development environment (IDE) and enter the following:

aws ecs create-cluster \
--cluster-name retable-ecs-cluster \
--region us-east-1

You should receive output similar to the following:

{
  "cluster": {
    "clusterArn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:cluster/retable-ecs-cluster",
    "clusterName": "retable-ecs-cluster",
    "status": "ACTIVE",
    "registeredContainerInstancesCount": 0,
    "runningTasksCount": 0,
    "pendingTasksCount": 0,
    "activeServicesCount": 0,
    "statistics": [],
    "tags": [],
    "settings": [
      {
        "name": "containerInsights",
        "value": "disabled"
      }
    ],
    "capacityProviders": [],
    "defaultCapacityProviderStrategy": []
  }
}

B. Create an IAM role

1. Open the IAM console at https://console.aws.amazon.com/iam/. In the navigation pane, choose Roles and then choose Create role.

2. Choose the AWS service role type, and then choose Systems Manager. Choose the Systems Manager use case and then Next: Permissions.

3. In the Attached permissions policy section, search for and select the AmazonSSMManagedInstanceCore and AmazonEC2ContainerServiceforEC2Role policies and then choose Next: Review.

4. For Add tags (optional), specify any custom tags to associate with the policy and then choose Next: Review.

5. For Role name, enter ECSAnywhereRole and optionally you can edit the description. Review your role information and then choose Create role.

6. Perform a search for the ECSAnywhereRole and then select it to view the role details.

7. On the Permissions tab, choose Attach policies. Search for the AmazonSSMManagedInstanceCore policy, select it, and then choose Attach policy.

Create a required IAM role “ecsAnywhereRole” for remote instances by following the instructions in IAM permissions for Amazon ECS Anywhere or the following steps:

Step 2: Prepare the infrastructure nodes

In this walkthrough, you use one virtual machine running your managed infrastructure. You can use any type of infrastructure that can run any of the operating systems supported by ECS Anywhere. This may include a physical, on-premises server.

A. Register your virtual machines

You can follow the instructions to learn more about how to register an external instance to an AWS ESC cluster.Alternately, to register both virtual machines using the AWS Management Console, perform the following instructions:

1. Open the Amazon ECS console.

2. From the navigation bar, select the Region to use.

3. In the navigation pane, choose Clusters.

4. On the Clusters page, choose awsmpdemo-ecs-cluster to register your external instance to.

5. Choose the ECS Instances tab, then choose Register external instances.

6. On the Step 1: External instances activation details page, complete the following steps.

   • For Activation key duration (in days), enter the number of days that the activation key remains active. After the number of days you entered passes, the key no longer works when registering an external instance.

   • For Number of instances, enter the number of external instances that you want to register to your cluster with the activation key.

   • For Instance role, choose the IAM role to associate with your external instances. We will use the IAM role ecsAnywhereRole created in previous step.

   • Choose Next step.

On the Step 2: Register external instances page, copy the registration command. This command should be run on each virtual machine that you prepared.

B. Verify the ECS agent is running on virtual machine

Once the installation completed, to verify if the ECS agent is running on virtual machine, run the following command in your virtual machine’s console.

 docker ps

You should receive output similar to the following example output:

root@onprem-vm1:/home/awsmpdemo# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f17dd66f70f6 amazon/amazon-ecs-agent:latest "/agent" 2 minutes ago Up 2 minutes (healthy) ecs-agent

On the AWS Management Console, navigate to the ECS Instances tab. Your remote instance should appear there.

A. Create an S3 bucket

Once the Amazon ECS cluster and external worker nodes are up and running, you can deploy the AWS Marketplace container product. Your IAM user must have appropriate permission to execute the following commands.

To create new S3 bucket, run the following command in your terminal. Be sure to give the bucket an appropriate name. See Bucket naming rules for more information on naming S3 buckets:

aws s3 mb s3://bucket-name --region us-east-1

Next, you must enable server-side encryption to secure your bucket by entering the following command in terminal:

aws s3api put-bucket-encryption \
    --bucket bucket-name \
    --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'

B. Set up required IAM role for executing ECS task

Create ecsTaskExecutionRole by following the instructions at Creating the task execution IAM role.

C. Set up IAM role with required permission for container product to perform actions on your behalf

1. Create an ecsPixyzTaskRole by running the following command in your terminal:

aws iam create-role \
--region us-east-1 \
--role-name ecsRetableTaskRole \
--assume-role-policy-document file://iam-task-execution-role.json

aws iam attach-role-policy \

--region us-east-1 \

--role-name ecsPixyzTaskRole \

--policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage

2. Create an inline policy document named AWSMP-Pixyz-task-role-policy.json. This allows read and writes to S3 objects of the S3 bucket created in step A. It also allows Amazon ECS’s task to perform necessary API calls to the AWS Marketplace metering service. See Amazon S3: Allows read and write access to objects in an S3 bucket and AWS managed policy: AWSMarketplaceMeteringRegisterUsage for more information on how to create the IAM policy document.

3. Attach the inline policy to your IAM role ecsRetableTaskRole by running the following command in your terminal:

aws iam put-role-policy \

--region us-east-1 \

--role-name ecsRetableTaskRole \

--policy-name ExamplePolicy \

--policy-document file://AWSMP-Retable-task-role-policy.json

4. Attach the AWSMarketplaceMeteringRegisterUsage managed policy to the IAM role ecsRetableTaskRole by running the following command in your terminal:

aws iam attach-role-policy \

--region us-east-1 \

--role-name ecsRetableTaskRole \

--policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage

D. Create ECS task definition file to deploy the container product on your ECS Anywhere cluster

See Retable's usage instructions for more information on the command arguments being passed to the container.

1. Create a task definition file.

To create an example task definition for ECS Anywhere, run the following command in your terminal, replacing <aws-account-id> with your actual AWS account ID and <bucket-name> with the name you chose in step A. You can follow the instructions here to find your AWS Account ID.

The requiresCompatibilities value must be EXTERNAL to allow the task to be deployed on external instances.

{
    "family": "retable-app"
    "containerDefinitions": [
        {
            "name": "api",
            "image": "709825985650.dkr.ecr.us-east-1.amazonaws.com/retable/rtbl-api:latest",
            "cpu": 3072,
            "memoryReservation": 7168,
            "portMappings": [
                {
                    "name": "4000",
                    "containerPort": 4000,
                    "hostPort": 4000,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environmentFiles": [],
            "healthCheck": {
                "command": [
                    "CMD-SHELL",
                    "curl -f http://localhost:4000/ || exit 1"
                ],
                "interval": 30,
                "timeout": 5,
                "retries": 3
            },
        },
        {
            "name": "ui",
            "image": "709825985650.dkr.ecr.us-east-1.amazonaws.com/retable/rtbl-ui:latest",
            "cpu": 512,
            "memoryReservation": 512,
            "portMappings": [
                {
                    "name": "3000",
                    "containerPort": 3000,
                    "hostPort": 3000,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environmentFiles": [],            
            "healthCheck": {
                "command": [
                    "CMD-SHELL",
                    "curl -f http://localhost:3000/ || exit 1"
                ],
                "interval": 30,
                "timeout": 5,
                "retries": 3
            },
        },
        {
            "name": "socket",
            "image": "709825985650.dkr.ecr.us-east-1.amazonaws.com/retable/rtbl-socket:latest",
            "cpu": 512,
            "memoryReservation": 512,
            "portMappings": [
                {
                    "name": "4500",
                    "containerPort": 4500,
                    "hostPort": 4500,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": false,
            "dependsOn": [
                {
                    "containerName": "api",
                    "condition": "HEALTHY"
                }
            ],
            "healthCheck": {
                "command": [
                    "CMD-SHELL",
                    "curl -f http://localhost:4500/ || exit 1"
                ],
                "interval": 30,
                "timeout": 5,
                "retries": 3
            },
        }
    ],
    "executionRoleArn": "arn:aws:iam::aws-account-id:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "EXTERNAL"
    ],
    "cpu": "4096",
    "memory": "8192",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    }
}

2. Register the task definition to your AWS account. To do that, enter the following command in your terminal:

aws ecs register-task-definition \
--region us-east-1 \
--cli-input-json file://AWSMP-Retable-task-execution.json

3. Run the registered task definition using the following command in your terminal:

aws ecs run-task \
--region us-east-1 \
--cluster retable-ecs-cluster \
--launch-type EXTERNAL \
--task-definition retable-app:1

You can monitor the progress in the Amazon ECS console until it runs successfully. My task is running successfully, as shown by the Desired task status of Running in the Tasks tab of my Amazon ECS console.