AWS Elasticsearch

Cloud.gov offers aws-elasticsearch as a service hosted in AWS Elasticsearch and Opensearch.

Plans

Plan Name	Instance Type	Description	Number of nodes
es-dev	t3.small.search	Single data node for non-prod use only	1
es-medium	c5.large.search	3 Primary and 2 Data node cluster	5
es-medium-ha	c5.large.search	3 Primary and 4 Data node cluster	7
es-large	c5.xlarge.search	3 Primary and 2 Data node cluster	5
es-large-ha	c5.xlarge.search	3 Primary and 4 Data node cluster	7
es-xlarge	c5.2xlarge.search	3 Primary and 2 Data node cluster	5
es-xlarge-ha	c5.2xlarge.search	3 Primary and 4 Data node cluster	7
es-2xlarge-gp	m5.2xlarge.search	3 Primary and 2 Data node cluster	5
es-2xlarge-gp-ha	m5.2xlarge.search	3 Primary and 4 Data node cluster	7
es-4xlarge-gp	m5.4xlarge.search	3 Primary and 2 Data node cluster	5
es-4xlarge-gp-ha	m5.4xlarge.search	3 Primary and 4 Data node cluster	7
es-12xlarge-gp	m5.12xlarge.search	3 Primary and 2 Data node cluster (by request)	5
es-12xlarge-gp-ha	m5.12xlarge.search	3 Primary and 4 Data node cluster (by request)	7

Service name is aws-elasticsearch for all plans.

Pricing

AWS Elasticsearch instances and storage will consume Cloud.gov resource credits; more information is available on the pricing page.

When to use

This service is geared toward applications that need to provide search capability, or interact with indexed data. Although Elasticsearch is often used as part of the ELK logging stack (Elasticsearch + Logstash + Kibana), this service does not include Kibana, and isn't suitable as a component of a logging solution. To find out more about logging on the Cloud.gov platform, please see the section on logs.

Create an instance

To create a service instance run the following command:

cf create-service aws-elasticsearch {service-plan} {service-name}

For example, to create a new instance using the es-medium plan named my-elastic-service, you would enter the following at the cf CLI:

cf create-service aws-elasticsearch es-medium my-elastic-service

Note: AWS Elasticsearch creation times will vary and is outside of Cloud.gov's control. AWS says approximately 15-30 mins per node.

Setting optional parameters

name	required	description	example
advanced_options	false	map for advanced options	see below
elasticsearchVersion	false	Specifies a supported major version in search (must be in "")	OpenSearch_3.7
log_publishing	false	Map of Cloudwatch log types to enable/disable (see Logging)	{"error_logs": true}

These are the current supported major versions for Elasticsearch version:

• OpenSearch_1.3
• OpenSearch_2.17
• OpenSearch_3.7

If you don't specify any option, currently OpenSearch_3.7 is used by default.

If you want to specify the Elasticsearch version:

cf create-service aws-elasticsearch \
    ${SERVICE_PLAN_NAME} \
    ${SERVICE_NAME} \
    -c '{"elasticsearchVersion": "OpenSearch_2.17"}'

Advanced Options

These are advanced tuning options that can have significant performance or behavior effects on your cluster. They are specified as key/value pairs under the advanced_options map in the core parameters. Note: although these all represent numbers, they are all specified as strings. Additionally, although they are dotted, they are not nested keys.

Name	description	default
indices.fielddata.cache.size	percentage of JVM heap allocated to field data	"20"
indices.query.bool.max_clause_count	maximum number of clauses allowed in a Lucene boolean query	"1024"

Here are some examples of how to set these values:

cf create-service aws-elasticsearch es-medium my-es-service-1 \
    -c '{"advanced_options": {"indices.fielddata.cache.size": "21"}}'

cf create-service aws-elasticsearch es-medium my-es-service-2 \
    -c '{"advanced_options": {"indices.query.bool.max_clause_count": "1025"}}'

cf create-service aws-elasticsearch es-medium my-es-service-3 \
    -c '{"advanced_options": {"indices.query.bool.max_clause_count": "1025", "indices.fielddata.cache.size": "21"}}'

Note - if you are using the cf CLI utility on Windows, see the examples section of the Cloud Foundry documentation for specific formatting of parameters.

Shard/replica configuration for high availability

The medium and large plans are mainly geared toward customers that need a single index and have coverage with 2 data nodes. For customers wanting more coverage and more indexes, the medium-ha nd large-ha plans scale the cluster to 4 data nodes to offer high availability (HA).

For additional information on configuring your service for high availability when using anything other than the es-dev plan, you can refer to this article from the Elasticsearch documentation for an overview of how clusters work.

AWS specific information can be found in the AWS Developer Guide. Note - AWS recently changed the name of this service to AWS OpenSearch.

Update an instance

Setting optional parameters

name	required	description	example
elasticsearchVersion	false	Target engine version for an in-place upgrade (see supported list above)	OpenSearch_3.7
volume_type	false	Specifies the EBS storage volume type. Valid options: gp3.	gp3
log_publishing	false	Map of Cloudwatch log types to enable/disable (see Logging)	{"error_logs": true}

Changing instance plans

You can move an existing instance to a different plan in place:

cf update-service ${SERVICE_NAME} -p ${NEW_SERVICE_PLAN_NAME}

A plan change resizes your existing domain rather than creating a new one. Only in-place upgrades are supported, so the broker checks the requested change before calling AWS and returns an error explaining the problem if the change isn't allowed. Rejected requests are refused immediately and leave your instance untouched, still running on its current plan.

The rules are:

• The new plan must be the same size or larger. Downgrading to a smaller plan isn't supported.
• The new plan can add data nodes but not remove them. The -ha plans run 4 data nodes and the non-HA plans run 2, so you can move from a non-HA plan to its -ha counterpart (or to a larger -ha plan). Going the other way is rejected, because removing data nodes would discard the data they hold.
• es-dev can't be scaled up. es-dev runs a single data node in one availability zone, while every other plan runs multiple data nodes across two, This plan is for development only and is not upgradeable.
• Only one change at a time. A plan change can't be combined with an engine version upgrade in the same command. Make them as separate cf update-service calls, waiting for each to finish.

Some examples:

From	To	Allowed?	Why
es-medium	es-large	Yes	larger plan, both 2 data nodes
es-medium-ha	es-large-ha	Yes	larger plan, both 4 data nodes
es-medium	es-medium-ha	Yes	same size, adding 2 data nodes
es-medium	es-large-ha	Yes	larger plan, adding 2 data nodes
es-large	es-medium	No	smaller plan
es-large-ha	es-large	No	removes 2 data nodes
es-medium-ha	es-large	No	removes 2 data nodes
es-dev	es-medium	No	es-dev runs in a single zone

For reference, this is the size order the rule uses for the 2-data-node plans, smallest to largest. The 4-data-node -ha plans follow the same order among themselves:

es-dev
es-medium
es-large
es-xlarge
es-2xlarge-gp
es-4xlarge-gp
es-12xlarge-gp

To make a change the broker won't allow — removing data nodes (moving from an -ha plan back to a non-HA plan), scaling up from es-dev, or moving to a smaller plan — create a new instance on the plan you want and move your data to it using the manual snapshot process described in Managing backups below.

warning

A plan change triggers an AWS blue/green deployment and may cause some amount of downtime while the domain is resized. AWS resizes can take 30 minutes or more depending on plan size and how much data you have. Plan the change for a low-traffic window, and check cf service ${SERVICE_NAME} for update succeeded before assuming it's done.

Note that plan storage sizes differ, and disk can only grow. If the new plan has a larger volume size your instance's storage grows to match it; if the new plan has a smaller volume size your existing storage is left as-is rather than being shrunk.

Upgrading the engine version

You can upgrade an existing instance to a newer OpenSearch or Elasticserach version in-place. To initiate an upgrade:

cf update-service ${SERVICE_NAME} \
  -c '{"elasticsearchVersion": "OpenSearch_3.7"}'

Please take note of the following rules and recommendations when upgrading the engine version for an instance:

• Version upgrades may not be combined with other modifications to the instance (such as volume_type, advanced_options, or a plan change with -p). Wait for the upgrade to finish before making additional changes.
• AWS only allows upgrades along supported version paths (ex: OpenSearch_2.17 -> OpenSearch_3.7). Attempting to upgrade to an unsupported target version or downgrading will result in an error. See the AWS OpenSearch upgrade documentation for the full list of supported upgrade paths.
• Elasticsearch_7.10 is retired. If your instance still runs it, upgrade to a supported OpenSearch version. AWS may require you to move through intermediate versions rather than jumping straight to OpenSearch_3.7, so check the supported upgrade paths in the AWS documentation linked above and run each step as a separate cf update-service, waiting for each to finish.

Upgrading to OpenSearch_3.7 on a domain that started as Elasticsearch

An index keeps a record of the engine that created it, and OpenSearch 3.x will not open an index that was created by Elasticsearch. This applies even if the domain has already been upgraded through OpenSearch 1.x and 2.x — the indexes carried along by those upgrades are still Elasticsearch-created indexes.

Before upgrading to OpenSearch_3.7, while still on OpenSearch_2.17, convert any Elasticsearch-era indexes to OpenSearch format by reindexing them into new indexes (or by restoring a snapshot into new indexes), then switch your application and any aliases over to the new indexes and delete the old ones. See the AWS OpenSearch upgrade documentation for the prerequisites and current guidance.

The upgrade pre-check AWS runs before the upgrade starts is the usual place this surfaces: the upgrade fails and your domain stays on its current version.

Updating storage volume type

To update an existing Elasticsearch/Opensearch instance to use gp3 storage volumes:

cf update-service ${SERVICE_NAME} \
  -c '{"volume_type": "gp3"}'

Updating the volume type to gp3 for your Elasticsearch/Opensearch instance will trigger a blue/green deployment and some amount of downtime while the instance updates.

Logging

Your instance can publish logs for monitoring cluster operations and troubleshooting issues. Enable them with cf create-service or cf update-service:

cf update-service ${SERVICE_NAME} \
  -c '{"log_publishing": {"error_logs": true, "search_slow_logs": true, "index_slow_logs": true}}'

key	description
error_logs	Engine error / application logs.
search_slow_logs	Search queries that exceed the given threshold. Requires threshold configuration - see below.
index_slow_logs	Index queries that exceed the given threshold. Requires threshold configuration - see below.
audit_logs	Tracks user activity, cluster access, data modifications, etc. Only supported on plans with Node-to-node encryption and Encryption at rest.

Slow logs require thresholds

Enabling search_slow_logs or index_slow_logs only turns on publishing to the logs platform; these logs will not produce anything until you configure the thresholds for these logs. You can update the threshold settings via the domain's REST API using the same AWS credentials that your app uses to access the domain.

See OpenSearch slow logs docs for more details.

Connecting to your service instance

If you need to directly access your service instance from your local environment, please read this knowledge base article for more information.

Managing space egress to connect to your service instance

If you are connecting to a service instance for the first time time, you'll need to update your space egress setting or you'll have connection refused errors.

note

By default, all spaces have their space egress set to closed which means any requests to the open internet or our brokered services will be blocked.

To allow connections from your space to your provisioned service, use the cf bind-security-group command. E.g., for your org, <ORG>, and space, <SPACE>, run:

cf bind-security-group trusted_local_networks_egress <ORG> --space <SPACE>

Managing backups

By default AWS will create hourly snapshot backups of your Elasticsearch service instance and retain those snapshots for 14 days - 336 total snapshots are retained. These snapshots are only accessible to cluster restore purposes only and can not be exported.

For customers that would like to import or export their Elasticsearch data, this can be done following the AWS Elasticsearch Manual Snapshot Process

As part of the process of using manual snapshots, customers will need to assign an existing S3 bucket and the broker will provide a AWS policy ARN to use with the manual S3 snapshot process

• Create an instance of the S3 service:

  cf create-service s3 basic my-s3-bucket

• Create a service key to access S3 credentials:

  cf create-service-key my-s3-bucket my-key

• Get your s3 bucket name:

  s3_credentials=$(cf service-key my-s3-bucket my-key | tail -n +3)
  s3_bucket=$(echo "${s3_credentials}" | jq -r '.credentials.bucket')

• Assign your new bucket to your Elasticsearch instance:

  cf create-service-key my-elastic-service my-key -c '{"bucket":"${s3_bucket}"}'

• Find your AWS ARN for your AWS policy to reference for your backup:

  es_arn=$(cf service-key my-elastic-service my-key | tail -n +3)
  snapshotRoleARN=$(echo "${es_arn}" | jq -r '.credentials.snapshotRoleARN')

Once you have your s3 bucket connected and have the snapshotRoleARN you can then inside your application connect to the AWS ES host and register your s3 repository endpoint and then perform your snapshot export/import operations using AWS signed HTTP headers. Due to the nature of AWS Signature Calculations there is no direct way with curl to perform these operations and it's best left to client libraries in a programing language your applications are written in.

Cloud.gov does offer a code sample repository on Github - aws-elasticsearch-example that shows an example in Python on how to interact with the new ES service using signed headers. Our customers are encouraged to submit PRs of other examples to share with fellow customers.

Encryption

Every non-dev AWS Elasticsearch instance configured through Cloud.gov is encrypted at rest. We use the industry standard AES-256 encryption algorithm to encrypt your data on the server that hosts your AWS Elasticsearch instance.

Service Name	Plan Name	Encryption at Rest
aws-elasticsearch	es-dev	No
aws-elasticsearch	es-medium	Yes
aws-elasticsearch	es-medium-ha	Yes

Note: If you have access to larger service plans, they will mirror same settings as es-medium or es-medium-ha.

Rotating credentials

You can rotate credentials by creating a new instance and deleting the existing instance. If this is not an option, email support@cloud.gov to request rotating the credentials manually.

The broker in GitHub

You can find the broker here: https://github.com/cloud-gov/aws-broker.