🔜AWS Marketplace Usage Instructions

This page serves as a comprehensive guide for setting up and installing Retable, a Unix-based application for data management. It outlines the system prerequisites, and installation steps for modules.

Contents

• Introduction

• Prerequisites

• Installation

  • API Module

  • Socket Module

  • UI Module

• AWS ECS Installation

Introduction

Retable is designed to operate exclusively in UNIX environments. All commands provided here have been tested on Unix-based EC2 instances, regardless of the specific OS chosen.

The Retable application comprises three main modules: API, UI, and Socket. Ensure all prerequisites are met before starting the application. Note that the API module is resource-intensive and requires the most allocation. Therefore, allocate a larger portion of your resources to the API module. The sequence for starting the modules should begin with the API, which depends on PostgreSQL and Redis connections.

Prerequisites

Database

A running PostgreSQL Database (minimum version 13) is required. For a small team, at least 1 CPU core and 2 GB RAM should be sufficient. If usage increases or performance issues arise, consider upgrading the server specifications accordingly.. The app uses two databases: one for meta-information and the other for storing table data.

Database Creation Options:

• Manual Database Creation:

  • Users can create the required databases manually using the following commands below:

  • In this case, the database names specified in the API environment file must match the manually created ones.
  • During initialization, the API will verify the existence of these databases. If they exist, it will proceed to create the necessary tables, indexes, and procedures automatically.

CREATE DATABASE rtbl_meta;
CREATE DATABASE rtbl_data;

• Automatic Database Creation:

  • If users do not create the databases manually, the API will create them during initialization based on the names specified in the environment file.
  • All required tables, indexes, and procedures will be generated automatically.

For both methods, users can define custom indexes, procedures, and additional configurations as needed.

Database Connection & Security Configuration:

• Specify the database connection details (user, password, database names, port, etc.) in the API environment file. More details can be found in the API Module Environment File section.
• The API module requires a TCP connection to the database server. Ensure that your security group allows traffic between the API and the database on the specified port.

Redis (Optional)

A running Redis Server (minimum version 6) is optional and required only for users who want to use collaborative functionalities and scheduled automations.

Redis is mainly used for the socket module to enable collaborative functionalities and for handling scheduled automations. If you want these features, you need a running Redis server and must configure it in both the API and Socket module’s environment files.

The API and Socket modules need TCP connection to the Redis server. Configure your security group to allow traffic between the API and Redis on the specified port.

Attachments

To store attachments, set up an AWS S3 Bucket. The API module requires the Bucket Region, Access Key, and Bucket Name, which should be configured in the API environment file. Ensure that proper access permissions are set for the bucket. More information is available in the API Module Environment File section.

The attachment limit per file is 100MB. If you exceed this limit, please contact us.

Creating an S3 Bucket

First, create an S3 bucket by following these instructions. After creating the attachment bucket, you must create an IAM role to manage your uploads.

Creating an S3 Policy

Retable requires permissions to upload attachments to your S3 bucket. To grant access, you need to create an S3 policy. Follow these steps to create the policy:

1

Go to the IAM Service in the AWS Console, select Policies from the left pane, and click Create policy.

2

Select S3 as the service

3

Under Actions, specify the allowed actions. Use the Filter Actions search bar to find and select PutObject, GetObject, and DeleteObject.

4

Under Resources, click Add ARNs. In the modal that appears, enter your bucket name in the Resource bucket name field and select the Any object name checkbox. Then, click Add ARNs and close the modal.

5

Click Next to proceed. Give the policy a descriptive name and, optionally, a description.

6

Click Create Policy to finalize the process.

Creating an IAM S3 User

After creating an S3 policy, you need to attach it to an IAM user. Follow these steps to create the IAM user:

1

Go to the IAM Service and select Users under the IAM Resources section.

2

Click Create User. On the next page, enter a name for the IAM user (e.g., s3-user) and click Next.

3

Permissions section, choose Attach policies directly. Search for the policy you created earlier, select the checkbox next to it, and click Next.

4

Review the details, and if everything is correct, click Create User.

Creating an Access Key

To generate an access key for the newly created IAM user, follow these steps:

1

Open the IAM Service, click Users in the left pane, and select the newly created user from the list.

2

On the user details page, click Create access key.

3

Choose Application running on an AWS compute service and check the confirmation box.

4

Optionally, add a description tag, then click Create access key.

5

On the final page, copy the Access Key and Secret Access Key. Ensure that you store the Secret Access Key securely, as it will not be shown again.

These keys are required for the API module environment file.

E-Mails

An SMTP server is necessary for sending emails, which is crucial for inviting users to your app. Ensure that the SMTP server details are correctly configured.

Installation

1. Prepare your database: Ensure your PostgreSQL database is set up and ready. More information.
2. Create the environment file (app_api_env):

   • Create a file named app_api_env in your project directory.

   • Add the required environment variables in the format KEY=VALUE, each on a new line.

Api Module Environment File:

Key	Description
ADMIN_USER_EMAIL	Email address of the admin user. This user will be created automatically when Product run
ADMIN_USER_PASSWORD	Password of the admin user
AWS_ECS_REGION	AWS region where ECS task is running (e.g., eu-west-2)
DB_NAME	The name of the meta database, which stores metadata
DB_DATA_NAME	The name of the data database, which stores retables table data
DB_HOST	Hostname of the Postgres Database Server
DB_PORT	Port of the Database Server
DB_USER	Username of the Database User
DB_PWD	Password of the Database User
REDIS_HOST	Hostname of the Redis Server
REDIS_PORT	Port number of the Redis Server
REDIS_PASSWORD	Password of the Redis Server
BASE_URL	Base URL to reach the whole application (e.g., https://yourdomain.retable.io
SHARE_URL	[will be filled]
S3_ACCESS_KEY	Your AWS Access Key
S3_SECRET_KEY	Your AWS Secret Access Key
S3_BUCKET_REGION	Your AWS S3 Bucket Region (ex: eu-west-1)
AWS_BUCKET_S3	Your AWS S3 Bucket name
META_BASE_URL	Base Endpoint to reach the API
EMAIL_SMTP	The SMTP server address used for sending emails
EMAIL_PORT	The port number used to connect to the SMTP server
EMAIL_AUTH_USER	The email used for authentication with the SMTP server
EMAIL_SEND_FROM	The default email address from which emails will be sent

Example format of the environment file:

ADMIN_USER_EMAIL=[e-mail]
ADMIN_USER_PASSWORD=!.!-RetableRocks-!.!

DB_HOST=localhost
DB_PORT=5432
DB_USER=postgres
DB_PWD=super_secret_pwd

REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=super_secret_pwd2

BASE_URL=https:\/\/retable.yourdomain.io
SHARE_URL=https:\/\/retable.yourdomaion.io\/v1\/go

AWS_BUCKET_UID=bucket_uid
AWS_BUCKET_PWD=bucket_pwd
AWS_BUCKET_REGION=eu-west-1
AWS_BUCKET_S3=retable_aws_s3_bucket

META_BASE_URL=https:\/\/retable-api.yourdomaion.io

Socket Module

1. Ensure your Redis is ready:

   • More information.
2. Create the environment file (app_socket_env):

   • Create a file named app_socket_env in your project directory.

   • Add the required environment variables to this file. Each variable should be on a new line in the format KEY=VALUE.

Socket Module Environment File:

Key	Description
REDIS_HOST	Hostname of the Redis
REDIS_PORT	Port number of the Redis Server
REDIS_PASSWORD	Password of the Redis Server
PORT	Port of your Socket Http & Websocket Server
META_API_BASE_ENDPOINT	Base URL of your API Module

Example format of the environment file:

REDIS_HOST=myredis.host
REDIS_PORT=6379
REDIS_PASSWORD=redisserverpass

PORT=4500

META_API_BASE_ENDPOINT=https:\/\/retable-api.yourdomaion.io

UI Module

1. Ensure your Api and Socket is ready:

   • For Api Module
   • For Socket Module
2. Create the environment file (app_ui_env):

   • Create a file named app_ui_env in your project directory.

   • Add the required environment variables to this file. Each variable should be on a new line in the format KEY=VALUE.

UI Module Environment File:

Key	Description
REACT_APP_API_PATH	Api Module Base Enpdoint
REACT_APP_BASE_URL	Base URL that you access your application
REACT_APP_SHARE_URL	Same with the REACT_APP_BASE_URL
REACT_APP_SOCKET_PATH	Socket Module Base Endpoint

Example format of the environment file:

REACT_APP_API_PATH=https:\/\/retable-api.yourdomaion.io\/v1
REACT_APP_BASE_URL=https:\/\/yourdomain.io
REACT_APP_SHARE_URL=https:\/\/yourdomain.io
REACT_APP_SOCKET_PATH=https:\/\/retable-socket.yourdomain.io

After creating these environment files, upload them to an S3 bucket. They will be required for the environmentFiles field when defining an ECS Task Definition.

AWS Elastic Container Service (ECS) Installation

Here’s how to get started with ECS Anywhere. You first create a new Amazon ECS cluster with two external worker nodes by installing Amazon ECS agents, System Manager agents, and Docker on virtual machines (VMs). Second, you create an Amazon S3 bucket to store input and output files for the ECS task and attachment files. Finally, you create AWS Identity and Access Management (IAM) roles with appropriate permissions and task definitions. This is to deploy AWS Marketplace container product on created ECS Anywhere cluster with a external worker node.

1. Subscribe to Retable product. This will incur software costs to your AWS account. You can review the product pricing at Retable – Pricing before subscribing.
2. Create IAM role.

3. Create Task definition.

4. Create Amazon ECS cluster and run the task definition on that Amazon ECS cluster.

Attention: Pre-install the following tools on remote node:

• AWS CLI
• Curl

Solution walkthrough

This solution walkthrough has the following steps:

1. Prepare the AWS resources.

2. Prepare the infrastructure node.

3. Deploy the container product.

Step 1: Prepare the AWS resources

In this step, you create an ECS Anywhere cluster to host the remote node and the required AWS IAM roles for the remote node and executing tasks.

A. Create an ECS Anywhere cluster using AWS CLI

To create an ECS Anywhere cluster, execute this command in your CLI:

aws ecs create-cluster \
--cluster-name retable-ecs-cluster \
--region us-east-1

You should receive output similar to the following:

{
  "cluster": {
    "clusterArn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:cluster/retable-ecs-cluster",
    "clusterName": "retable-ecs-cluster",
    "status": "ACTIVE",
    "registeredContainerInstancesCount": 0,
    "runningTasksCount": 0,
    "pendingTasksCount": 0,
    "activeServicesCount": 0,
    "statistics": [],
    "tags": [],
    "settings": [
      {
        "name": "containerInsights",
        "value": "disabled"
      }
    ],
    "capacityProviders": [],
    "defaultCapacityProviderStrategy": []
  }
}

B. Create an IAM role

Create a required IAM role “ecsAnywhereRole” for remote instances by following the instructions in IAM permissions for Amazon ECS Anywhere or the following steps:

1

Open the IAM console at https://console.aws.amazon.com/iam/. In the navigation pane, choose Roles and then choose Create role.

2

Choose the AWS service role type, and then choose Systems Manager. Choose the Systems Manager use case and then Next: Permissions.

3

permissions policy section, search for and select the AmazonSSMManagedInstanceCore and AmazonEC2ContainerServiceforEC2Role policies and then choose Next: Review.

4

For Add tags (optional), specify any custom tags to associate with the policy and then choose Next: Review.

5

For Role name, enter ECSAnywhereRole and optionally you can edit the description. Review your role information and then choose Create role.

6

Perform a search for the ECSAnywhereRole and then select it to view the role details.

7

On the Permissions tab, make sure that AmazonSSMManagedInstanceCore policy is added.

Step 2: Prepare the infrastructure node

In this walkthrough, you use a virtual machine running your managed infrastructure. You can use any type of infrastructure that can run any of the operating systems supported by ECS Anywhere. This may include a physical, on-premises server.

A. Register your virtual machines

You can follow the instructions to learn more about how to register an external instance to an AWS ESC cluster. Alternately, to register virtual machine using the AWS Management Console, perform the following instructions:

1

Open the Amazon ECS console.

2

From the navigation bar, select the Region to use.

3

In the navigation pane, choose Clusters.

4

On the Clusters page, choose retable-ecs-cluster to register your external instance to.

5

Choose the Infastructure tab, then choose Register external instances on the Container Instances section.

6

On the Step 1: External instances activation details page, complete the following steps:

• For Activation key duration (in days), enter the number of days that the activation key remains active. After the number of days you entered passes, the key no longer works when registering an external instance.

• For Number of instances, enter the number of external instances that you want to register to your cluster with the activation key.

• For Instance role, choose the IAM role to associate with your external instances. We will use the IAM role ecsAnywhereRole created in previous step.

• Choose Generate registration command.

  You should see a command similar to this:

  curl --proto "https" -o "/tmp/ecs-anywhere-install.sh" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh" && bash /tmp/ecs-anywhere-install.sh --region "us-east-1" --cluster "retable-ecs-cluster" --activation-id "<activation_id>" --activation-code "<activation_code>"

• Copy the generated registration command then run this command on your virtual machine that you prepared.

B. Verify the ECS agent is running on virtual machine

Once the installation completed, to verify if the ECS agent is running on virtual machine, run the following command in your virtual machine’s console.

docker ps

You should receive output similar to the following example output:

root@onprem-vm1:/home/awsmpdemo# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f17dd66f70f6 amazon/amazon-ecs-agent:latest "/agent" 2 minutes ago Up 2 minutes (healthy) ecs-agent

On the AWS Management Console, navigate to the Infrastructure tab. Your remote instance should appear on the Container instances section.

A. Set up required IAM role for executing ECS task

Create ecsTaskExecutionRole by following the instructions at Creating the task execution IAM role.

B. Attach the required permissions

1. Metering Usage policy

   Attach the AWSMarketplaceMeteringFullAccess managed policy to the IAM role ecsTaskExecutionRole by running the following command in CLI:

   aws iam attach-role-policy \
   --region us-east-1 \
   --role-name ecsTaskExecutionRole \
   --policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringFullAccess

2. S3 Get Policy

   Attach the AmazonS3ReadOnlyAccess managed policy to the IAM role ecsTaskExecutionRole by running the following command in CLI:

   aws iam attach-role-policy \
   --region us-east-1 \
   --role-name ecsTaskExecutionRole \
   --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

3. Metering Usage

   aws iam attach-role-policy \
   --region us-east-1 \
   --role-name ecsTaskExecutionRole \
   --policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage

C. Create ECS task definition file to deploy the container product on your ECS Anywhere cluster

See Retable's usage instructions for more information on the command arguments being passed to the container.

1. Create a task definition file.

To create an task definition for ECS Anywhere save this as file called AWSMP-Retable-task-execution.json replacing <aws-account-id> with your actual AWS account ID and <bucket-name> with the name you chose for the environment files. You can follow the instructions here to find your AWS Account ID.

The requiresCompatibilities value must be EXTERNAL to allow the task to be deployed on external instances.

{
  "family": "retable-app",
  "containerDefinitions": [
    {
      "name": "api",
      "image": "709825985650.dkr.ecr.us-east-1.amazonaws.com/retable/retable-api:1.7.2",
      "portMappings": [
        {
          "containerPort": 4000,
          "hostPort": 4000,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "environmentFiles": [
        {
          "value": "arn:aws:s3:::bucket_name/folder/api-aws.docker.env",
          "type": "s3"
        }
      ],
      "healthCheck": {
        "command": [
          "CMD-SHELL",
          "curl -f http://localhost:4000/ || exit 1"
        ],
        "interval": 30,
        "timeout": 5,
        "retries": 3
      }
    },
    {
      "name": "ui",
      "image": "709825985650.dkr.ecr.us-east-1.amazonaws.com/retable/retable-ui:2.0.1",
      "portMappings": [
        {
          "containerPort": 3000,
          "hostPort": 3000,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "environmentFiles": [
        {
          "value": "arn:aws:s3:::bucket_name/folder/ui-aws.docker.env",
          "type": "s3"
        }
      ],
      "healthCheck": {
        "command": [
          "CMD-SHELL",
          "curl -f http://localhost:3000/ || exit 1"
        ],
        "interval": 30,
        "timeout": 5,
        "retries": 3
      }
    },
    {
      "name": "socket",
      "image": "709825985650.dkr.ecr.us-east-1.amazonaws.com/retable/retable-socket:0.9.0",
      "portMappings": [
        {
          "containerPort": 4500,
          "hostPort": 4500,
          "protocol": "tcp"
        }
      ],
      "essential": false,
      "environmentFiles": [
        {
          "value": "arn:aws:s3:::bucket_name/folder/socket-aws.docker.env",
          "type": "s3"
        }
      ],
      "dependsOn": [
        {
          "containerName": "api",
          "condition": "HEALTHY"
        }
      ],
      "healthCheck": {
        "command": [
          "CMD-SHELL",
          "curl -f http://localhost:4500/ || exit 1"
        ],
        "interval": 30,
        "timeout": 5,
        "retries": 3
      }
    }
  ],
  "executionRoleArn": "arn:aws:iam::aws-account-id:role/ecsTaskExecutionRole",
  "networkMode": "bridge",
  "requiresCompatibilities": [
    "EXTERNAL"
  ],
  "cpu": "2048",
  "memory": "7673",
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  }
}

1. Register the task definition to your AWS account. To do that, enter the following command in your terminal:

aws ecs register-task-definition \
--region us-east-1 \
--cli-input-json file://AWSMP-Retable-task-execution.json

1. Run the registered task definition using the following command in your terminal:

aws ecs run-task \
--region us-east-1 \
--cluster retable-ecs-cluster \
--launch-type EXTERNAL \
--task-definition retable-app:1

You can monitor the progress in the Amazon ECS console until it runs successfully.