Bonjour,
As you’ve noticed I’m French, so forgive me if my English isn’t perfect. 😉
Since I joined this company I’ve been streamlining the way we use Amazon’s cloud infrastructure (AWS). Outlandish uses AWS to host a range of applications including ElasticSearch (ES) which we use for analysing large quantities of twitter data.
Instead of using AWS you can use Qbox which is a hosted ElasticSearch, it’s more expensive than AWS but you need to care less about managing your cluster.
On the internet, you can find lots of information on how to create an AWS instance and how to install ES but none really explain how to put ES on an AWS instance and running it as a service, that is to say when the instance launches. This is precisely what this post is about.
Steps to go through
Step 1 : Create an account on AWS, a keypair and a security group
You can find some useful documentation here :
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html
For the security group I advise you to use these rules :
Note that for ElasticSearch to work properly you need to allow inbound connections on ports 9200 and 9300.
The SSH rule is to allow you to connect to your instance and the HTTP one is to access to the web.
However, we got hacked pretty quickly when we opened up those ports to the whole world, you’d better restrict access to your instance as shown above.
Step 2 : Launch an AWS instance
What you have to do now is to connect yourself to the AWS console, you should see something like this :
1. Choose AMI
Choose a Amazon Linux AMI , you can’t miss it, it’s the first one that you see, the one with a yellow cube. All our servers are running this AMI (image), it’s quite similar to Redhat, so if you know about linux you should get use to it easily.
2. Choose Instance type
To start I suggest you choose the t2.micro or t1.micro (free tier eligible) but for the future you might want to choose another one according for what you want to use your instance. In order to decide yourself, you might want to have a look around here :
http://aws.amazon.com/blogs/aws/choosing-the-right-ec2-instance-type-for-your-application/
http://aws.amazon.com/ec2/instance-types/
3. Configure Instance
I’ve accidentaly terminated an instance, so I strongly recomend you to tick the ‘delete on termination’ box.
The other options aren’t necessary to configure if you just want to test.
4. Add Storage
As a test, you can leave the settings as default but if you want to store more data you might want to increase this value.
5. Tag Instance
Optional, if you’re interested in this, check this link :
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html
6. Configure Security Group
Select an existing security group
Then choose the securtiy group you’ve created in Step 1.
7. Review
Click on “ Launch”, a window will appear, you must select the keypair you’ve created in Step 1 and tick the small box otherwises you won’t be able to launch you’re instance.
Step 3 : Log into your instance
In the “Instances”section, select your instance and click on “Connect” and follow the instructions.
Step 4 : Install the latest version of ES as a service
Make sure you are root. (run the sudo su command)
Run the commands in the yum section :
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html
Then run these :
cd /etc/init.d/
chkconfig --add elasticsearch
/etc/init.d/elasticsearch start
Step 5 : Avoid future issues
We’ve experienced some issues such as having all the memory being used by an instance this was coming from the fact that the instance was creating some hprof files.
To avoid that run this command :
nano /usr/share/elasticsearch/bin/elasticsearch.in.sh
and put a “#” in front of the this line of the file :
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
Another one was some security problems which you can avoid with this command and by adding the second line at the beginning of the elasticsearch.yml file :
nano /etc/elasticsearch/elasticsearch.yml
script.disable_dynamic: true
In order that these changes apply, you must restart ES with this command :
service elasticsearch restart
Step 6 : Test ElasticSearch
Run :
curl -X GET http://localhost:9200/
If this command shows something as below then ES works fine.
Step 7 : Have fun configuring ElasticSearch as you like
You can use the readme file (located in /usr/share/elasticsearch/) to get started and have a look at these websites :
Please let me have any feedback on the above and put any questions in the comments below. I hope you find this helpful.
[Abi wrote this bit] We are always looking for new developers so if you’ve read this blog, found it helpful and are interested to find out more, get in touch, we are always up for a pint and a chat in the pub team@outlandish.com
