Recent Changes

Thursday, August 4

  1. page Setting Up EC2 edited ... listen_addresses = '*' That'll let Postgres listen to connections. UPDATE You may also hav…
    ...
    listen_addresses = '*'
    That'll let Postgres listen to connections.
    UPDATE
    You may also have to set up Postgres to trust local connections. I'm not sure if this is the best way to do it, but it's what worked for me.
    $server: vi /etc/postgresql/8.4/main/pg_hba.conf
    And change the 'md5' to 'trust' on the localhost domains
    # Database administrative login by UNIX sockets
    local all postgres trust
    # TYPE DATABASE USER CIDR-ADDRESS METHOD
    # "local" is for Unix domain socket connections only
    local all all trust
    # IPv4 local connections:
    host all all 127.0.0.1/32 trust #was md5
    # IPv6 local connections:
    host all all ::1/128 trust #was md5
    # Amazon private network
    host all all 10.0.0.0/8 md5
    and finally, restart postgres
    $server: /etc/init.d/postgresql restart

    Install GRASS
    It seems that the Django application doesn't work properly if you don't also install grass (which isn't part of the kitchen sink I guess. Maybe it's elsewhere in the kitchen). This will pop up once you do an address search. An error like "cannot import name SpatialRefSys"
    (view changes)
    10:34 am

Tuesday, July 19

  1. page Notes on review app edited The Reporter's Lab tool review site will fill a hole in the journalism ecosystem by testing open s…
    The Reporter's Lab tool review site will fill a hole in the journalism ecosystem by testing open source and commercial reporting and Web tools against real-world tasks and documents. For example:
    Test scraping tools against simple sites and those filled with javascript or logons.
    Test document analysis software against a variety of documents held in many different forms, such as single pdfs of e-mail collections and harvested web pages.
    Evaluate speech and video recognition tools
    Quickly show reporters what small, free tools to work with pdfs or documents might give them.
    Separately, we'll create a task library that includes each iteration of the sample documents and a detailed description of the tasks we want to accomplish. The task library would be ideally linked into the database of reviews to let others use them in their own tests or to let reporters see exactly what they look like.
    This is the first project of the lab for several reasons, but the most important is that it will let us see what holes there are in the world of free and open source tools, helping drive our work in the future.
    Some of the elements that would be useful to be able to browse and search are:
    Cost / licensing
    Tested on which tasks / documents / data? (See related project, Task library)
    Installation required
    Programming required (none -> substantial)
    Results for each test
    Narrative for the blog posting.
    Advice to people who want to try it
    Type of task (eg, text analysis, audio records, scraping, visualization, etc.)
    Rating?
    Who did the testing
    Date(s) reviewed
    What version tested
    Op system, etc. environment tested.
    Stars? / Ratings for usability, accuracy, difficulty?
    It would be really useful to have a section of the database to track the reviews and reviewers. I already know of probably two dozen tools that I'd like to test, and it would helpful if I could enter them and assign them out to reviewers as I can. Parts of this section might not be published, but could include:
    Tool name
    Source / how to get it
    What it claims to do
    Who is reviewing it
    When assigned
    When due
    Completed when
    What it's good for (what tasks we'lll assign it)

    (view changes)
    7:27 am

Monday, July 18

  1. page Setting Up Your Mac edited ... Open your Terminal application (it's in Applications/Utilities). At the prompt type vi $ v…
    ...
    Open your Terminal application (it's in Applications/Utilities).
    At the prompt type
    vi$ vi .bash_profile
    (some instructions on how to use Vi if you're not familiar: http://www.infobound.com/vi.html -- all you really need to know is to hit the "i" key and start putting in the code you want. When done hit the escape key, and type ":wq" to save and quit. or ":q" to quit if you didn't change anything. or ":q!" to quit and not save your changes).
    I found a sample .bash_profile after some searching, here's what mine looks like (again, not sure if this is sane or necessary, but here you go):
    ...
    The last couple of lines I found I had to add after trying numerous times to get psycopg2 (required for Django) to work properly. An explanation can be found here: http://www.chrisbaltzer.com/post/6754911523/building-python-libs-with-xcode-4. I followed his instructions to also add a line to the /etc/sudoers file as well.
    I also edited the .bashrc file to play around with the colors of my terminal and change the way the prompt looks. Much of it isn't necessary, but if you do decide to change the colors also be sure to change the terminal preferences to have a black background with white text and something bright for your bold text.
    vi$ vi .bashrc
    and this is what I had at this point
    #paths
    ...
    At this point, I think I was set up to properly follow the Chicago Tribune's tutorial (one of many excellent pieces they put together) on setting things up for Django development: http://blog.apps.chicagotribune.com/2010/02/17/quick-install-pythonpostgis-geo-stack-on-snow-leopard/
    Here are the commands
    easy_install$ easy_install virtualenv
    easy_install

    $ easy_install
    virtualenvwrapper
    easy_install

    $ easy_install
    ipython
    This will install VirtualEnv, VirtualEnvWrapper, and an interactive python interpreter (the last of which I haven't really used yet -- you can tell I'm a novice). VirtualEnv is great for segregating your python libraries so you can run multiple versions of libraries against your various projects. Keeps things from conflicting.
    I also installed pip, yolk, and fabric. These may be better when installed directly into a virtual environment, I'm fuzzy on that. Regardless:
    sudo$ sudo easy_install pip
    sudo

    $ sudo
    pip install
    Fabric is used for deplyoment, yolk is used to see what libraries you have installed into your python environment.
    Next, you install brew, postgis, postgres, gdal, and git. Brew helps tremendously in installing packages.
    sudo$ sudo mkdir /usr/local
    sudo

    $ sudo
    chown -R 'whoami' /usr/local
    curl

    $ curl
    -L http://github.com/mxcl/homebrew/tarball/master
    installs brew
    brew$ brew install postgis
    brew

    $ brew
    install gdal --with-postgres
    brew

    $ brew
    install git
    installs the rest.
    Note: Here is where I ran into endless pain with GDAL. It kept asking for X11, which wasn't installed on my machine by default. Hopefully the earlier step in this guide will have fixed that problem.
    (view changes)
    12:39 pm
  2. page Setting Up EC2 edited ... EC2 initial setup There is an excellent tutorial for this here: http://www.robertsosinski.com…
    ...
    EC2 initial setup
    There is an excellent tutorial for this here: http://www.robertsosinski.com/2008/01/26/starting-amazon-ec2-with-mac-os-x/
    ...
    here: https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key.
    After

    After
    creating your
    ...
    them secure.
    You can then grab the EC2 command line tools from here: http://aws.amazon.com/developertools/351?_encoding=UTF8&jiveRedirect=1. Extract them somewhere and keep track where you put them.
    Now you'll want to create a directory to hold your keys and tools...
    ...
    so, open the id_rsa.pub file on your local machine and copy the contents:
    #copy what appears and close vi
    $localmachine:$local: vi ~/.ssh/id_rsa.pub
    on the server do this:
    #paste the public key into a new-line in this file
    $ec2-instance:$server: vi .ssh/authorized_keys
    Now if you you 'exit' the server, you should be able to log back in by just doing this:
    $ ssh -l newsapps xxx-instance-xxx.compute-1.amazonaws.com
    ...
    It seems that the Django application doesn't work properly if you don't also install grass (which isn't part of the kitchen sink I guess. Maybe it's elsewhere in the kitchen). This will pop up once you do an address search. An error like "cannot import name SpatialRefSys"
    $server: sudo apt-get install grass grass-doc
    ...
    hang ups:
    Don't stupidly remove the [] around your EC2 instance name, it's supposed to be a list.
    The S3 bucket is just that, the bucket name, nothing else. The fabfile.py file seems to indicate there's more to it.
    (view changes)
    12:38 pm
  3. page Setting Up EC2 edited ... Now if you you 'exit' the server, you should be able to log back in by just doing this: $ ssh…
    ...
    Now if you you 'exit' the server, you should be able to log back in by just doing this:
    $ ssh -l newsapps xxx-instance-xxx.compute-1.amazonaws.com
    Next Steps
    Now that you have a server up and running, you can deploy your Hello, Newsroom app to it. Following the instructions is pretty easy, and I only ran into a couple of hang-ups.
    Postgres Fix
    First, it seems that the AMI I launched didn't have Postgres set up to actually allow connections to it. I ran into a server error while trying to access the application:
    "could not connect to server: connection refused (0x0000274D/10061). Is the server running on host 'localhost' and accepting TCP/IP connection on port 5432?"
    The fix to this is to allow connections...
    $server: sudo su - postgres
    $server: vi /etc/postgresql/8.4/main/postgresql.conf
    And change
    #listen_addresses = ''
    listen_addresses = '*'
    That'll let Postgres listen to connections.
    Install GRASS
    It seems that the Django application doesn't work properly if you don't also install grass (which isn't part of the kitchen sink I guess. Maybe it's elsewhere in the kitchen). This will pop up once you do an address search. An error like "cannot import name SpatialRefSys"
    $server: sudo apt-get install grass grass-doc
    Smaller hang ups:
    Don't stupidly remove the [] around your EC2 instance name, it's supposed to be a list.
    The S3 bucket is just that, the bucket name, nothing else. The fabfile.py file seems to indicate there's more to it.
    Having edited the fabfile, the settings, and the Apache config, you can...
    $local: fab staging master setup deploy
    And you should be able to go to your instance URL (something like http://ec2-72-44-53-181.compute-1.amazonaws.com) and it'll hopefully work!

    (view changes)
    12:37 pm
  4. page Setting Up EC2 edited ... This will take a few minutes, but when it's done you'll have a server running. The Cloud Comma…
    ...
    This will take a few minutes, but when it's done you'll have a server running. The Cloud Commander tells you how to login, but it has never worked in my case. Here's what I had to do to log on to the server:
    $ ssh -i /path/to/pem/yourkeypair.pem newsapps@ec2-xx-xxx-xx-xx.compute-1.amazonaws.com
    You should be on the server at this point. That wasn't so hard, was it?? *falls over in exhaustion*
    The next step I took was to install some SSH keys onto the server so I could login without having to specify the .pem file every time. If you followed the last tutorial you should have created some keys for Github, so you'll already have something to use.
    so, open the id_rsa.pub file on your local machine and copy the contents:
    #copy what appears and close vi
    $localmachine: vi ~/.ssh/id_rsa.pub
    on the server do this:
    #paste the public key into a new-line in this file
    $ec2-instance: vi .ssh/authorized_keys
    Now if you you 'exit' the server, you should be able to log back in by just doing this:
    $ ssh -l newsapps xxx-instance-xxx.compute-1.amazonaws.com

    (view changes)
    12:19 pm
  5. page Setting Up EC2 edited ... "Key Pair" is the name of the key-pair you created, you can see a list in the AWS co…
    ...
    "Key Pair" is the name of the key-pair you created, you can see a list in the AWS console under the EC2 section in the left rail
    "Security Group" is in the AWS console under the EC2 section as well. Will probably just be "default"
    Once you've filled all of this out, it'll generate a script used for actually launching the servers. You'll also want to copy your .pem file of the key you specified during setup into the assets folder cloud-commander generates.
    $ cp ~/.ec2/yourkey.pem my_servers/assets/yourkey.pem
    Cloud Commander will move it to your instance for you.
    Now, launch the server. The Cloud Commander downloads a number of possible options from a recipe github repository, but you can also create your own. Here I am launching one of the built-in recipes:
    $ cd my_servers
    $ cloud-commander start newsapps-kitchensink
    This will take a few minutes, but when it's done you'll have a server running. The Cloud Commander tells you how to login, but it has never worked in my case. Here's what I had to do to log on to the server:
    $ ssh -i /path/to/pem/yourkeypair.pem newsapps@ec2-xx-xxx-xx-xx.compute-1.amazonaws.com

    (view changes)
    12:11 pm
  6. page Setting Up EC2 edited ... You can then grab the EC2 command line tools from here: http://aws.amazon.com/developertools/3…
    ...
    You can then grab the EC2 command line tools from here: http://aws.amazon.com/developertools/351?_encoding=UTF8&jiveRedirect=1. Extract them somewhere and keep track where you put them.
    Now you'll want to create a directory to hold your keys and tools...
    cd$ cd ~
    mkdir

    $ mkdir
    .ec2
    cd

    $ cd
    .ec2
    open

    $ open
    .
    This will pop open a finder window on a Mac. Copy the lib + bin folders from your extracted EC2 tools into this folder. Also copy the two .pem key files you downloaded from Amazon into here. You can close the finder window at this point.
    Next, you'll want to edit your .bash_profile to insert some environment variables
    vi$ vi ~/.bash_profile
    And add.
    # Setup Amazon EC2 Command-Line Tools
    ...
    export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/
    And either close / re-open your finder, or
    source$ source ~/.bash_profile
    Make a KeyPair

    At this point you can create another key-pair for logging onto an instance you create. Note that this can also be done through the AWS console under the EC2 section, here: https://console.aws.amazon.com/ec2/ (it's in the left-rail towards the bottom). If you do it that way, you'll want to download the .pem file it generates and save it to your .ec2 directory (you'll have to save it elsewhere and copy it into that folder using the Finder trick above or the command line).
    ec2-add-keypair$ ec2-add-keypair ec2-keypair
    It'll spit out a private RSA key. Copy the contents and paste them into a new document called ec2-keypair. Save that into your .ec2 folder.
    You'll want to modify the permissions of the file so it's only visible to your user (it may not work otherwise):
    #where 'ec2-keypair' is the file you saved the private key to.
    chmod$ chmod 600 ec2-keypair
    Make a User
    Next you'll want to create a user in the system so you can get access keys (seems like there are a ton of different keys here, it's exhausting). You can do this through the web console here: https://console.aws.amazon.com/iam/home
    It'll spit out a CSV containing an Access Key and a Secret Key for every user you create. You will need to save this in a safe place, you'll only ever be able to get the secret keys once.
    Open Some Ports

    You're pretty close to having everything set-up on the EC2 side at this point. There's one more step needed to open the ports of your "security group" so you can actually ssh into them (or view them over the web). I BELIEVE you can do this step now, but you may have to wait until you actually launch your first instance. It appears to be a one-time thing for each security group in your AWS account.
    #open port 22 for ssh
    ec2-authorize$ ec2-authorize default -p
    #open port 80 for web access
    ec2-authorize$ ec2-authorize default -p 80
    You should be good to go to actually launch an instance now. For this is used the Trib's Cloud-Commander
    Cloud-Commander
    I wanted to get things up fast and still learn enough that I could make changes in the future. I could have launched the Tribunes already-prepared AMI (an Amazon image for generating servers), but I wanted to try their Cloud-Commander instead.
    For whatever reason, neither the easy_install nor pip versions of the cloud-commander work, so I installed from source: https://github.com/newsapps/cloud-commander
    They say it makes the most sense to first do this in a virtual environment. So, assuming you've already got their Hello Newsroom Django application already set-up to run under a hello_newsroom virtual environment, here's the steps you'd want to take:
    $ workon hello_newsroom
    $ cd /your/project/directory
    #actually get the code and run it.
    $ git clone https://github.com/newsapps/cloud-commander.git
    $ cd cloud-commander
    $ python setup.py install
    This should install the cloud-commander. Next you'll want to set it up.
    #do this in your project directory
    $ cloud-commander setup my_servers
    It's going to ask you a bunch of questions, some of which aren't super-clear
    AWS access key comes from IAM in console
    AWS secret key comes from CSV when user is created (in the CSV)
    S3 Bucket comes from creating an S3 bucket (super-simple to do in the AWS console. This is just the name of the bucket you created, nothing special)
    Default Region and Zone in my case was us-east-1
    "Key Pair" is the name of the key-pair you created, you can see a list in the AWS console under the EC2 section in the left rail
    "Security Group" is in the AWS console under the EC2 section as well. Will probably just be "default"

    (view changes)
    12:02 pm

More