Setting up on EC2


This will be an explanation of what I had to get through to run the Chicago Tribune's "Hello, Newsroom" application on EC2, including getting EC2 set up on your machine. This explanation assumes you followed the Setting Up Your Mac wiki to get things like your GitHub account created and setup on your machine.

--

EC2 initial setup


There is an excellent tutorial for this here: http://www.robertsosinski.com/2008/01/26/starting-amazon-ec2-with-mac-os-x/

The above is slightly dated, the new link to download your security credentials (The x.509 certificate) is here: https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key.

After creating your Amazon Web Services (AWS) account, go to the above URL and download your X.509 Certificate and your RSA Private Key. These are used by the EC2 command-line tools to access your AWS account. You'll need to save these and keep them secure.

You can then grab the EC2 command line tools from here: http://aws.amazon.com/developertools/351?_encoding=UTF8&jiveRedirect=1. Extract them somewhere and keep track where you put them.

Now you'll want to create a directory to hold your keys and tools...

$ cd ~
$ mkdir .ec2
$ cd .ec2
$ open .
This will pop open a finder window on a Mac. Copy the lib + bin folders from your extracted EC2 tools into this folder. Also copy the two .pem key files you downloaded from Amazon into here. You can close the finder window at this point.

Next, you'll want to edit your .bash_profile to insert some environment variables

$ vi ~/.bash_profile
And add.

# Setup Amazon EC2 Command-Line Tools
export EC2_HOME=~/.ec2
export PATH=$PATH:$EC2_HOME/bin
#the EC2 Private Key, it begins with "pk-"
export EC2_PRIVATE_KEY=$EC2_HOME/[[YOUR_PRIVATE_KEY]]
#the EC2 X.509 Certificate, begins with "cert-"
export EC2_CERT=$EC2_HOME/[[YOUR CERT]]
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/
And either close / re-open your finder, or

$ source ~/.bash_profile

Make a KeyPair


At this point you can create another key-pair for logging onto an instance you create. Note that this can also be done through the AWS console under the EC2 section, here: https://console.aws.amazon.com/ec2/ (it's in the left-rail towards the bottom). If you do it that way, you'll want to download the .pem file it generates and save it to your .ec2 directory (you'll have to save it elsewhere and copy it into that folder using the Finder trick above or the command line).

$ ec2-add-keypair ec2-keypair
It'll spit out a private RSA key. Copy the contents and paste them into a new document called ec2-keypair. Save that into your .ec2 folder.

You'll want to modify the permissions of the file so it's only visible to your user (it may not work otherwise):

#where 'ec2-keypair' is the file you saved the private key to.
$ chmod 600 ec2-keypair

Make a User


Next you'll want to create a user in the system so you can get access keys (seems like there are a ton of different keys here, it's exhausting). You can do this through the web console here: https://console.aws.amazon.com/iam/home

It'll spit out a CSV containing an Access Key and a Secret Key for every user you create. You will need to save this in a safe place, you'll only ever be able to get the secret keys once.


Open Some Ports


You're pretty close to having everything set-up on the EC2 side at this point. There's one more step needed to open the ports of your "security group" so you can actually ssh into them (or view them over the web). I BELIEVE you can do this step now, but you may have to wait until you actually launch your first instance. It appears to be a one-time thing for each security group in your AWS account.

#open port 22 for ssh
$ ec2-authorize default -p 22
#open port 80 for web access
$ ec2-authorize default -p 80

You should be good to go to actually launch an instance now. For this is used the Trib's Cloud-Commander

Cloud-Commander


I wanted to get things up fast and still learn enough that I could make changes in the future. I could have launched the Tribunes already-prepared AMI (an Amazon image for generating servers), but I wanted to try their Cloud-Commander instead.

For whatever reason, neither the easy_install nor pip versions of the cloud-commander work, so I installed from source: https://github.com/newsapps/cloud-commander

They say it makes the most sense to first do this in a virtual environment. So, assuming you've already got their Hello Newsroom Django application already set-up to run under a hello_newsroom virtual environment, here's the steps you'd want to take:

$ workon hello_newsroom
$ cd /your/project/directory
 
#actually get the code and run it.
$ git clone https://github.com/newsapps/cloud-commander.git
$ cd cloud-commander
$ python setup.py install
This should install the cloud-commander. Next you'll want to set it up.

#do this in your project directory
$ cloud-commander setup my_servers
It's going to ask you a bunch of questions, some of which aren't super-clear

  • AWS access key comes from IAM in console
  • AWS secret key comes from CSV when user is created (in the CSV)
  • S3 Bucket comes from creating an S3 bucket (super-simple to do in the AWS console. This is just the name of the bucket you created, nothing special)
  • Default Region and Zone in my case was us-east-1
  • "Key Pair" is the name of the key-pair you created, you can see a list in the AWS console under the EC2 section in the left rail
  • "Security Group" is in the AWS console under the EC2 section as well. Will probably just be "default"


Once you've filled all of this out, it'll generate a script used for actually launching the servers. You'll also want to copy your .pem file of the key you specified during setup into the assets folder cloud-commander generates.

$ cp ~/.ec2/yourkey.pem my_servers/assets/yourkey.pem
Cloud Commander will move it to your instance for you.

Now, launch the server. The Cloud Commander downloads a number of possible options from a recipe github repository, but you can also create your own. Here I am launching one of the built-in recipes:

$ cd my_servers
$ cloud-commander start newsapps-kitchensink
This will take a few minutes, but when it's done you'll have a server running. The Cloud Commander tells you how to login, but it has never worked in my case. Here's what I had to do to log on to the server:

$ ssh -i /path/to/pem/yourkeypair.pem newsapps@ec2-xx-xxx-xx-xx.compute-1.amazonaws.com
You should be on the server at this point. That wasn't so hard, was it?? *falls over in exhaustion*

The next step I took was to install some SSH keys onto the server so I could login without having to specify the .pem file every time. If you followed the last tutorial you should have created some keys for Github, so you'll already have something to use.

so, open the id_rsa.pub file on your local machine and copy the contents:

#copy what appears and close vi
$local: vi ~/.ssh/id_rsa.pub

on the server do this:

#paste the public key into a new-line in this file
$server: vi .ssh/authorized_keys
 
Now if you you 'exit' the server, you should be able to log back in by just doing this:

$ ssh -l newsapps xxx-instance-xxx.compute-1.amazonaws.com
 


Next Steps




Now that you have a server up and running, you can deploy your Hello, Newsroom app to it. Following the instructions is pretty easy, and I only ran into a couple of hang-ups.

Postgres Fix


First, it seems that the AMI I launched didn't have Postgres set up to actually allow connections to it. I ran into a server error while trying to access the application:

"could not connect to server: connection refused (0x0000274D/10061). Is the server running on host 'localhost' and accepting TCP/IP connection on port 5432?"
The fix to this is to allow connections...

$server: sudo su - postgres
$server: vi /etc/postgresql/8.4/main/postgresql.conf
 
And change
#listen_addresses = ''
listen_addresses = '*'
 
That'll let Postgres listen to connections.

UPDATE

You may also have to set up Postgres to trust local connections. I'm not sure if this is the best way to do it, but it's what worked for me.

$server: vi /etc/postgresql/8.4/main/pg_hba.conf
And change the 'md5' to 'trust' on the localhost domains

# Database administrative login by UNIX sockets
local   all         postgres                          trust
 
# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
 
# "local" is for Unix domain socket connections only
local   all         all                               trust
# IPv4 local connections:
host    all         all         127.0.0.1/32          trust #was md5
# IPv6 local connections:
host    all         all         ::1/128               trust #was md5
# Amazon private network
host    all         all         10.0.0.0/8            md5
 
and finally, restart postgres

$server: /etc/init.d/postgresql restart
 




Install GRASS

It seems that the Django application doesn't work properly if you don't also install grass (which isn't part of the kitchen sink I guess. Maybe it's elsewhere in the kitchen). This will pop up once you do an address search. An error like "cannot import name SpatialRefSys"

$server: sudo apt-get install grass grass-doc

Smaller hang ups:
  • Don't stupidly remove the [] around your EC2 instance name, it's supposed to be a list.
  • The S3 bucket is just that, the bucket name, nothing else. The fabfile.py file seems to indicate there's more to it.

Having edited the fabfile, the settings, and the Apache config, you can...

$local: fab staging master setup deploy
And you should be able to go to your instance URL (something like http://ec2-72-44-53-181.compute-1.amazonaws.com) and it'll hopefully work!