AWS Marketplace Usage Instructions
This page serves as a comprehensive guide for setting up and installing Retable, a Unix-based application for data management. It outlines the system prerequisites, and installation steps for modules.
Last updated
This page serves as a comprehensive guide for setting up and installing Retable, a Unix-based application for data management. It outlines the system prerequisites, and installation steps for modules.
Last updated
API Module
Socket Module
UI Module
Retable is designed to operate exclusively in UNIX environments. All commands provided here have been tested on Unix-based EC2 instances, regardless of the specific OS chosen.
The Retable application comprises three main modules: API, UI, and Socket. Ensure all prerequisites are met before starting the application. Note that the API module is resource-intensive and requires the most allocation. Therefore, allocate a larger portion of your resources to the API module. The sequence for starting the modules should begin with the API, which depends on PostgreSQL and Redis connections.
A running PostgreSQL Database (minimum version 13) is required. For a small team, at least 1 CPU core and 2 GB RAM should be sufficient. If usage increases or performance issues arise, consider upgrading the server specifications accordingly.. The app uses two databases: one for meta-information and the other for storing table data.
Manual Database Creation:
Users can create the required databases manually using the following commands below:
During initialization, the API will verify the existence of these databases. If they exist, it will proceed to create the necessary tables, indexes, and procedures automatically.
Automatic Database Creation:
All required tables, indexes, and procedures will be generated automatically.
For both methods, users can define custom indexes, procedures, and additional configurations as needed.
Database Connection & Security Configuration:
The API module requires a TCP connection to the database server. Ensure that your security group allows traffic between the API and the database on the specified port.
A running Redis Server (minimum version 6) is optional and required only for users who want to use collaborative functionalities and scheduled automations.
The API and Socket modules need TCP connection to the Redis server. Configure your security group to allow traffic between the API and Redis on the specified port.
To store attachments, set up an AWS S3 Bucket. The API module requires the Bucket Region, Access Key, and Bucket Name, which should be configured in the API environment file. Ensure that proper access permissions are set for the bucket. More information is available in the API Module Environment File section.
Retable requires permissions to upload attachments to your S3 bucket. To grant access, you need to create an S3 policy. Follow these steps to create the policy:
Go to the IAM Service in the AWS Console, select Policies from the left pane, and click Create policy.
Select S3 as the service
Under Actions, specify the allowed actions. Use the Filter Actions search bar to find and select PutObject, GetObject, and DeleteObject.
Under Resources, click Add ARNs. In the modal that appears, enter your bucket name in the Resource bucket name field and select the Any object name checkbox. Then, click Add ARNs and close the modal.
Click Next to proceed. Give the policy a descriptive name and, optionally, a description.
Click Create Policy to finalize the process.
After creating an S3 policy, you need to attach it to an IAM user. Follow these steps to create the IAM user:
Go to the IAM Service and select Users under the IAM Resources section.
Click Create User. On the next page, enter a name for the IAM user (e.g., s3-user) and click Next.
Permissions section, choose Attach policies directly. Search for the policy you created earlier, select the checkbox next to it, and click Next.
Review the details, and if everything is correct, click Create User.
To generate an access key for the newly created IAM user, follow these steps:
Open the IAM Service, click Users in the left pane, and select the newly created user from the list.
On the user details page, click Create access key.
Choose Application running on an AWS compute service and check the confirmation box.
Optionally, add a description tag, then click Create access key.
On the final page, copy the Access Key and Secret Access Key. Ensure that you store the Secret Access Key securely, as it will not be shown again.
An SMTP server is necessary for sending emails, which is crucial for inviting users to your app. Ensure that the SMTP server details are correctly configured.
Create the environment file (app_api_env
):
Create a file named app_api_env
in your project directory.
Add the required environment variables in the format KEY=VALUE
, each on a new line.
ADMIN_USER_EMAIL
Email address of the admin user. This user will be created automatically when Product run
ADMIN_USER_PASSWORD
Password of the admin user
AWS_ECS_REGION
AWS region where ECS task is running (e.g., eu-west-2)
DB_NAME
The name of the meta database, which stores metadata
DB_DATA_NAME
The name of the data database, which stores retables table data
DB_HOST
Hostname of the Postgres Database Server
DB_PORT
Port of the Database Server
DB_USER
Username of the Database User
DB_PWD
Password of the Database User
REDIS_HOST
Hostname of the Redis Server
REDIS_PORT
Port number of the Redis Server
REDIS_PASSWORD
Password of the Redis Server
BASE_URL
SHARE_URL
[will be filled]
S3_ACCESS_KEY
Your AWS Access Key
S3_SECRET_KEY
Your AWS Secret Access Key
S3_BUCKET_REGION
Your AWS S3 Bucket Region (ex: eu-west-1)
AWS_BUCKET_S3
Your AWS S3 Bucket name
META_BASE_URL
Base Endpoint to reach the API
EMAIL_SMTP
The SMTP server address used for sending emails
EMAIL_PORT
The port number used to connect to the SMTP server
EMAIL_AUTH_USER
The email used for authentication with the SMTP server
EMAIL_SEND_FROM
The default email address from which emails will be sent
Example format of the environment file:
Ensure your Redis is ready:
Create the environment file (app_socket_env
):
Create a file named app_socket_env
in your project directory.
Add the required environment variables to this file. Each variable should be on a new line in the format KEY=VALUE
.
REDIS_HOST
Hostname of the Redis
REDIS_PORT
Port number of the Redis Server
REDIS_PASSWORD
Password of the Redis Server
PORT
Port of your Socket Http & Websocket Server
META_API_BASE_ENDPOINT
Base URL of your API Module
Example format of the environment file:
Ensure your Api and Socket is ready:
Create the environment file (app_ui_env
):
Create a file named app_ui_env
in your project directory.
Add the required environment variables to this file. Each variable should be on a new line in the format KEY=VALUE
.
REACT_APP_API_PATH
Api Module Base Enpdoint
REACT_APP_BASE_URL
Base URL that you access your application
REACT_APP_SHARE_URL
Same with the REACT_APP_BASE_URL
REACT_APP_SOCKET_PATH
Socket Module Base Endpoint
Example format of the environment file:
After creating these environment files, upload them to an S3 bucket. They will be required for the environmentFiles field when defining an ECS Task Definition.
Create IAM role.
Create Task definition.
Create Amazon ECS cluster and run the task definition on that Amazon ECS cluster.
Attention: Pre-install the following tools on remote node:
Curl
This solution walkthrough has the following steps:
Prepare the AWS resources.
Prepare the infrastructure node.
Deploy the container product.
In this step, you create an ECS Anywhere cluster to host the remote node and the required AWS IAM roles for the remote node and executing tasks.
A. Create an ECS Anywhere cluster using AWS CLI
To create an ECS Anywhere cluster, execute this command in your CLI:
You should receive output similar to the following:
B. Create an IAM role
Choose the AWS service role type, and then choose Systems Manager. Choose the Systems Manager use case and then Next: Permissions.
permissions policy section, search for and select the AmazonSSMManagedInstanceCore and AmazonEC2ContainerServiceforEC2Role policies and then choose Next: Review.
For Add tags (optional), specify any custom tags to associate with the policy and then choose Next: Review.
For Role name, enter ECSAnywhereRole and optionally you can edit the description. Review your role information and then choose Create role.
Perform a search for the ECSAnywhereRole and then select it to view the role details.
On the Permissions tab, make sure that AmazonSSMManagedInstanceCore policy is added.
A. Register your virtual machines
From the navigation bar, select the Region to use.
In the navigation pane, choose Clusters.
On the Clusters page, choose retable-ecs-cluster to register your external instance to.
Choose the Infastructure tab, then choose Register external instances on the Container Instances section.
On the Step 1: External instances activation details page, complete the following steps:
For Activation key duration (in days), enter the number of days that the activation key remains active. After the number of days you entered passes, the key no longer works when registering an external instance.
For Number of instances, enter the number of external instances that you want to register to your cluster with the activation key.
For Instance role, choose the IAM role to associate with your external instances. We will use the IAM role ecsAnywhereRole
created in previous step.
Choose Generate registration command.
You should see a command similar to this:
curl --proto "https" -o "/tmp/ecs-anywhere-install.sh" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh" && bash /tmp/ecs-anywhere-install.sh --region "us-east-1" --cluster "retable-ecs-cluster" --activation-id "<activation_id>" --activation-code "<activation_code>"
Copy the generated registration command then run this command on your virtual machine that you prepared.
B. Verify the ECS agent is running on virtual machine
Once the installation completed, to verify if the ECS agent is running on virtual machine, run the following command in your virtual machineโs console.
You should receive output similar to the following example output:
On the AWS Management Console, navigate to the Infrastructure tab. Your remote instance should appear on the Container instances section.
A. Set up required IAM role for executing ECS task
B. Attach the required permissions
Metering Usage policy
Attach the AWSMarketplaceMeteringFullAccess managed policy to the IAM role ecsTaskExecutionRole by running the following command in CLI:
S3 Get Policy
Attach the AmazonS3ReadOnlyAccess managed policy to the IAM role ecsTaskExecutionRole by running the following command in CLI:
Metering Usage
C. Create ECS task definition file to deploy the container product on your ECS Anywhere cluster
See Retable's usage instructions for more information on the command arguments being passed to the container.
Create a task definition file.
The requiresCompatibilities value must be EXTERNAL to allow the task to be deployed on external instances.
Register the task definition to your AWS account. To do that, enter the following command in your terminal:
Run the registered task definition using the following command in your terminal:
You can monitor the progress in the Amazon ECS console until it runs successfully.
In this case, the database names specified in the must match the manually created ones.
If users do not create the databases manually, the API will create them during initialization based on the names specified in the .
Specify the database connection details (user, password, database names, port, etc.) in the API environment file. More details can be found in the API Module Environment File .
Redis is mainly used for the socket module to enable collaborative functionalities and for handling scheduled automations. If you want these features, you need a running Redis server and must configure it in both the and moduleโs environment files.
The attachment limit per file is 100MB. If you exceed this limit, please
First, create an S3 bucket by following these . After creating the attachment bucket, you must create an IAM role to manage your uploads.
These keys are required for the .
Prepare your database: Ensure your PostgreSQL database is set up and ready. .
Base URL to reach the whole application (e.g.,
.
Hereโs how to get started with ECS Anywhere. You first create a new Amazon ECS cluster with two external worker nodes by installing Amazon ECS agents, System Manager agents, and Docker on virtual machines (VMs). Second, you create an bucket to store input and output files for the ECS task and attachment files. Finally, you create roles with appropriate permissions and task definitions. This is to deploy AWS Marketplace container product on created ECS Anywhere cluster with a external worker node.
Subscribe to . This will incur software costs to your AWS account. You can review the product pricing at before subscribing.
Create a required IAM role โecsAnywhereRole
โ for remote instances by following the instructions in or the following steps:
Open the IAM console at . In the navigation pane, choose Roles and then choose Create role.
In this walkthrough, you use a virtual machine running your managed infrastructure. You can use any type of infrastructure that can run any of . This may include a physical, on-premises server.
You can follow the instructions to learn more about how to . Alternately, to register virtual machine using the AWS Management Console, perform the following instructions:
Open the .
Create ecsTaskExecutionRole by following the instructions at .
To create an task definition for ECS Anywhere save this as file called AWSMP-Retable-task-execution.json replacing <aws-account-id> with your actual AWS account ID and <bucket-name> with the name you chose for the environment files. You can follow the instructions .