Home / Blog / Creating an AWS instance running ElasticSearch

Creating an AWS instance running ElasticSearch

By Michael Ohl, 11 Aug 2014

Bonjour,

As you’ve noticed I’m French, so forgive me if my English isn’t perfect. ūüėČ

Since I joined this company I’ve been streamlining the way we use¬†Amazon’s cloud infrastructure¬†(AWS). Outlandish uses AWS to host a range of applications¬†including ElasticSearch¬†(ES)¬†which we use for analysing large quantities of¬†twitter¬†data.

Instead of using AWS you can use¬†Qbox¬†which is a hosted ElasticSearch, it’s more expensive than AWS but you need to care less about managing your cluster.

On the internet, you can find lots of information on how to create an AWS instance and how to install ES but none really explain how to put ES on an AWS instance and running it as a service, that is to say when the instance launches. This is precisely what this post is about.

Steps to go through

Step 1 : Create an account on AWS, a keypair and a security group

You can find some useful documentation here :

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html

For the security group I advise you to use these rules :

right rules

Note that for ElasticSearch to work properly you need to allow inbound connections on ports 9200 and 9300.

The SSH rule is to allow you to connect to your instance and the HTTP one is to access to the web.

However, we got hacked pretty quickly when we opened up those ports to the whole world, you’d better restrict access to your instance as shown above.

Step 2 : Launch an AWS instance

What you have to do now is to connect yourself to the AWS console, you should  see something like this :

AWS_console

1. Choose AMI

Choose a Amazon Linux AMI , you can’t miss it, it’s the first one that you see, the one with a ¬†yellow cube. All our servers ¬†are running this AMI (image), it’s quite similar to Redhat, so if you know about linux you should get use to it easily.

2. Choose Instance type

To  start  I suggest you choose the t2.micro or t1.micro (free tier eligible) but for the future you  might want to choose  another one  according for what you want to use your instance. In order to decide yourself, you might want to have a look around here :

http://aws.amazon.com/blogs/aws/choosing-the-right-ec2-instance-type-for-your-application/

http://aws.amazon.com/ec2/instance-types/

3. Configure Instance

I’ve accidentaly terminated an instance, so I strongly recomend you to¬†tick the ‘delete on termination’ box.

The other options aren’t necessary to configure if you just want to test.

4. Add Storage

As  a test, you can leave the settings as default but if you want to store more data you might want to increase this value.

5. Tag Instance

Optional, if you’re interested in this, check this link :

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html

6. Configure Security Group

 Select an existing security group

Then choose the securtiy group you’ve created in Step 1.

7. Review

Click on ‚Äú¬†Launch‚ÄĚ, a window will appear, you must select the keypair ¬†you’ve created in Step 1¬†and tick the small box otherwises you won’t be able to ¬†launch you’re instance.

 

Step 3 : Log into your instance

In the ‚ÄúInstances‚ÄĚsection, select your instance and click on “Connect” and follow the instructions.

connect

Step 4 : Install the latest version of ES as a service

Make sure you are root. (run the sudo su command)

Run the commands in the yum section :

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html

 

Then run these :

cd /etc/init.d/
chkconfig --add elasticsearch
/etc/init.d/elasticsearch start

 

Step 5 : Avoid future issues

We’ve experienced some issues such as having all the memory being used by an instance this was coming from the fact that the instance was creating some hprof files.

To avoid that run this command :

nano /usr/share/elasticsearch/bin/elasticsearch.in.sh

and put a “#” in front of the this line of the file :

JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"

 

Another one was some security problems which you can avoid with this command and by adding the second line at the beginning of the elasticsearch.yml file :

nano /etc/elasticsearch/elasticsearch.yml
script.disable_dynamic: true


In order that  these changes apply, you must restart  ES with this command :

service elasticsearch restart

Step 6 : Test ElasticSearch

Run :

curl -X GET http://localhost:9200/

If this command shows something as below then ES works fine.

output

 

Step 7 : Have fun configuring ElasticSearch as you like

You can use the readme file (located in /usr/share/elasticsearch/) to get started and have a look at these websites :

 

Please let me have any feedback on the above and put any questions in the comments below. I hope you find this helpful.

[Abi wrote this bit] We are always looking for new developers so if you’ve read this blog, found it helpful and are interested to find out more, get in touch, we are always up for a pint and a chat in the pub team@outlandish.com