The Curious Dev

Various programming sidetracks and shiny-object detours

Cloudifying MoinMoin

This post takes the Installing MoinMoin on Nginx and uWSGI post further by deploying the built instance of MoinMoin within an AutoScalingGroup (ASG).

Creating the AMI

From my previous post of we have come out with a running instance configured as desired. The only trouble with this situation is that it’s a snowflake.

First step to producing a server that can be repeatedly recreated is to take a snapshot of the instance, this is as simple as choosing so in the AWS console and giving it a name.

Create an Amazon Machine Image(AMI)

Take the time to describe your image so you can remember what it does down the track.

Describe your image

It doesn’t typically take very long, certainly not with these tiny instances with only 8GB of disk.

Pending image

You can then find the created image under the Images > AMIs menu in the console.

Pending image

This AMI ID is what we’ll use in the next step.

Templating with CloudFormation

So now we’ve got an AMI, the next step is to create a CloudFormation template that will produce an ASG for the MoinMoin application to be served out of.

First up, we create a skeleton of the template, with some metadata, parameters and resources:

{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description": "Template for the MoinMoin wiki at wiki.easyas.info.",
"Parameters" : {

},
"Resources": {

}

In the Parameters section I’ll add a few options that might be needed to be tweaked at deployment time. Properties such as the aforementioned AMI, the EC2 instance type and the VPC subnet to operate in. I’m cheating a little here by using a Security Group that I created manually for another project.

"Parameters" : {
  "BaseAMI": {
      "Description": "AMI to use for AutoScaling",
      "Type": "String",
      "Default": "ami-00bd6960"
  },
  "InstanceType": {
      "Description": "Instance Type",
      "Type": "String",
      "Default": "t2.nano"
  },
  "moinmoinSG": {
    "Description": "SecurityGroup to control access to instance",
    "Type": "String",
    "Default": "sg-0feddb69"
  },
  "moinmoinKeypair": {
    "Description": "The keypair used by instances within the ASG",
    "Type": "AWS::EC2::KeyPair::KeyName",
    "Default": "ec2-user-keys"
  },
  "vpcSubnet": {
    "Description": "VPC Subnet for all the things, for now",
    "Type": "String",
    "Default": "subnet-2653ec43"
  }
}

Next up is the meat of the template, Resources, where the actual ASG is declared along with the LaunchConfiguration that it needs to know how to fire up instances.

Here’s the LaunchConfiguration, where we start with the MoinMoin AMI, configure an 8GB disk, and then setup the UserData to ensure that MoinMoin is up and running as soon as the instance comes up:

"Resources": {
  "moinmoinLC": {
    "Type": "AWS::AutoScaling::LaunchConfiguration",
    "Properties": {
        "AssociatePublicIpAddress": "true",
        "BlockDeviceMappings": [
          {
            "DeviceName": "/dev/xvda",
            "Ebs": {
              "DeleteOnTermination": "true",
              "VolumeType": "gp2",
              "VolumeSize": "8"
            }
          }
        ],
        "ImageId": { "Ref" : "BaseAMI" },
        "KeyName": { "Ref" : "moinmoinKeypair" },
        "IamInstanceProfile": "moinmoin-role",
        "InstanceType": { "Ref" : "InstanceType" },
        "SecurityGroups": [ { "Ref" : "moinmoinSG" }  ],
        "UserData": {"Fn::Base64" : { "Fn::Join" : ["", [
          "#!/bin/sh", "\n",
          "yum update -y", "\n",
          "#TODO call script to update IP in Route53 config for wiki.easyas.info"
          "start moin", "\n",
          "service nginx start", "\n"
          ]]}
        }
    }
  }
}

An ASG is a really powerful way to provide reliability with minimal effort. The AutoScalingGroup is where we reference the above “moinmoinLC” LaunchConfiguration and declare the UpdatePolicy and the number of instances.

The UpdatePolicy is the way that CloudFormation knows how to transition from one version of a template to another. The DesiredCapacity is just that, how many instances we’d prefer to have. Having the MinSize and MaxSize both set to one simply ensures there’s always a new instance brought up if the running one were to fail with no more than one instance being provisioned at any one time.

"Resources": {
  "moinmoinASG": {
    "Type" : "AWS::AutoScaling::AutoScalingGroup",
    "Properties": {
      "DesiredCapacity": "1",
      "HealthCheckGracePeriod": 180,
      "HealthCheckType": "EC2",
      "LaunchConfigurationName": { "Ref": "moinmoinLC" },
      "MaxSize": "1",
      "MinSize": "1",
      "Tags": [
        {
          "Key": "Name",
          "Value": "moinmoin-wiki",
          "PropagateAtLaunch": true
        }
      ],
      "VPCZoneIdentifier": [ { "Ref": "vpcSubnet" } ]
    },
    "UpdatePolicy" : {
      "AutoScalingRollingUpdate" : {
        "MinInstancesInService" : "0",
        "MaxBatchSize" : "1",
        "WaitOnResourceSignals" : "true",
        "PauseTime" : "PT5M"
      }
    }
  }
}

So that completes the template and we can simply run that in the CloudFormation console, there are several options but here I just upload the file:

Choose your template file

Then we select the values for the parameters, most of which are conveniently prefilled by the default values of our template parameters:

Select parameter values

Add some tags, Name becomes the name of the provisioned instance(s):

Tagging the template

Review and confirm the template execution:

Review and confirm the template

If things go smoothly, you’ll be able to see the template execute and end up in a clean “CREATE_COMPLETE” state.

Successful template creation

You should then be able to go on over to the EC2 console and see the running instance:

The provisioned instance for the ASG

And of course, check out the site!

MoinMoin, cloudified!

As an alternative to using the AWS console, you can execute a CloudFormation template from the AWS CLI incredibly easily:

aws cloudformation create-stack --stack-name mystackname --template-url s3://mybucket/a.template.file.json --parameters s3://mybucket/my.parameters.file.json

Hope that helps demonstrate how easy it can be to fire up an Auto Scaled server with minimal fuss.

Installing MoinMoin on Nginx and uWSGI

MoinMoin is a great wiki having used it on and off over the years. This post demonstrates how to get MoinMoin up and running with Nginx and uWSGI.

Install nginx

Simply install via package:

yum install nginx -y

Then configure /etc/nginx/nginx.conf with a section similar to this before the existing default section:

server {
    server_name wiki.easyas.info;

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:///usr/local/local/share/moin/moin.sock;
        uwsgi_modifier1 30;
    }
}

I’m using wiki.easyas.info as the domain here, but this could just as easily be the IP of your instance.

Install uWSGI

uWSGI is the backend that MoinMoin prefers to run on, but you need a C compiler to install it, install GCC:

yum install gcc -y

Then, install uWSGI via pip:

pip install uwsgi

Which should allow one to run uwsgi --version but didn’t for me, it’s not critical however, we’ll be using the full path anyway.

Create the logging directory too, uwsgi might not run without it:

mkdir /var/log/uwsgi

Install MoinMoin

So with the basic infrastructure in place, we now install and configure MoinMoin.

Download the binary from http://static.moinmo.in/files/moin-1.9.8.tar.gz then extract it to somewhere useful i.e. /tmp/moin-1.9.8.:

cd /tmp
wget http://static.moinmo.in/files/moin-1.9.8.tar.gz
tar xvf moin-1.9.8.tar.gz

Install moinmoin by executing the setup.py script within the moin-* directory:

cd moin-*
python setup.py install --prefix=/usr/local

Copy /tmp/moin-1.9.8/wiki/server/moin.wsgi to /usr/local/local/share/moin/moin.wsgi.

Configure /usr/local/local/share/moin/moin.wsgi, somewhere after ‘import sys, os’, add these lines:

sys.path.insert(0, '/usr/local/local/lib/python2.7/site-packages/')
sys.path.insert(0, '/usr/local/local/share/moin/')

Next up, to get moinmoin up and running on uwsgi, we need to create the file /usr/local/local/share/moin/uwsgi.ini and populate it with this:

[uwsgi]
uid = nginx
gid = nginx
socket = /usr/local/local/share/moin/moin.sock
chmod-socket = 660
logto = /var/log/uwsgi/uwsgi.log

chdir = /usr/local/local/share/moin/
wsgi-file = moin.wsgi

master
workers = 3
max-requests = 200
harakiri = 30
die-on-term

Note that I’ve changed the user to ‘nginx’ to align with the Nginx account which simplifies permissions. Obviously some people aren’t fond of this idea and would rather run each service under their own account, but for my purposes of this wiki I’m happy enough.

Create the file /etc/init/moin.conf for the moin service:

description "moin uwsgi service"

start on runlevel [2345]
stop on runlevel [!2345]

chdir /usr/local/local/share/moin
exec /usr/local/bin/uwsgi /usr/local/local/share/moin/uwsgi.ini
respawn

Copy /tmp/moin-1.9.8/wiki/config/wikiconfig.py to /usr/local/local/share/moin/wikiconfig.py.

Now edit the file and update various settings accordingly, much can stay as default, but it’s likely you’ll at least want to change these ones:

  • sitename i.e. “easyAsWiki”
  • page_front_page i.e. “StartPage”
  • superuser i.e. [u”AdminUser”, ]
  • acl_rights_before i.e. [u”AdminUser:read,write,delete,revert,admin”]

Finally, change the ownership of various directories to play nicely as the nginx user:

chown nginx:nginx -R /usr/local/local/share/moin
chown nginx:nginx -R /var/log/nginx
chown nginx:nginx -R /var/log/uwsgi

So that’s about it for the setup, you can now start nginx service nginx start and start the uwsgi server start moin. Then navigate to the IP of your instance and you should see MoinMoin’s start page come up.

StartPage

Reference

Static Hosting With AWS

This site is hosted from an AWS S3 bucket and fronted by the AWS CloudFront CDN service. Continuing a recent theme about security, I figured I’d provide an updated guide to configuring S3 and CloudFront hosting with the additional angle of securing with TLS.

I briefly covered CDNs and CloudFront in easy wins for website performance. Essentially a CDN is a vast collection of servers distributed throughout the world in a way that they’re “close” to the consumers of the assets served. Most commonly this is done to save on latency of sending images/html/etc all the way around the world.

In this guide I’ll use my little play domain easyas.info to demonstrate. For the most part the AWS CLI client is a readily accessible way to do most things on AWS, but sometimes it’s just easier to drop into the web console, so I do a bit of both.

Hosting from a bucket [S3]

Create a bucket:

aws --region us-east-1 s3 mb s3://easyas.info

Configure the bucket for website serving:

aws --region us-east-1 s3 website s3://easyas.info/ --index-document index.html

I’ve not bothered at this point, but you could also specify an –error-document property at this point too, to serve up to the user if they have a bad URL or some other HTTP 4xx error code.

Next, add a bucket policy to allow visitors to view the site:

aws --region us-east-1 s3api put-bucket-policy --bucket easyas.info --policy file://easyas.info.policy

Here, the easyas.info.policy file contains:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadForGetBucketObjects",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::easyas.info/*"
        },
        {
            "Sid": "ListBucketObjects",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::easyas.info"
        }
    ]
}

There are many, many other options for website hosting on S3, but that is enough to get up and going.

Put something in your bucket, a basic web page to get going, here’s a starter:

<html>
<head><title>Hosted on S3!</title></head>
<body>
<h2>easyas.info</h2>
<ol>
  <li>S3</li>
  <li>Cloudfront</li>
  <li>Route53</li>
  <li>ACM</li>
</ol>
</body>
</html>

Throw that up into the bucket with something like this:

aws --region us-east-1 s3 cp index.html s3://easyas.info/index.html

Over in the AWS console, find your bucket and drill down to the Static Website Hosting > Enable website hosting section and put this XML in the Redirection Rules text area:

<RoutingRules>
    <RoutingRule>
        <Condition>
            <KeyPrefixEquals>/</KeyPrefixEquals>
        </Condition>
        <Redirect>
            <ReplaceKeyPrefixWith>/index.html</ReplaceKeyPrefixWith>
        </Redirect>
    </RoutingRule>
</RoutingRules>

This will ensure that if you’ve got sub-directories with index.html documents, they’ll be served up when you just have the directory name in the URL, thus https://easyas.info/about will work just as good as https://easyas.info/about/index.html.

Configuring a CDN [CloudFront]

Creating a CloudFront distribution requires quite a few configuration items, but most of them can just be left as default.

From the main screen of CloudFront, click the Create Distribution button.

Create Distribution

For the delivery method, we want to choose Web, so click the relevant Get Started button.

For the Origin Settings, in the Origin Domain Name box, choose your S3 bucket from the drop down (click in the text box). But don’t leave it as that, you’ll want to change the URL for the Origin Name and Path to be the full S3 Website Hosting URL, i.e. easyas.info.s3-website-us-east-1.amazonaws.com (rather than easyas.info.s3.amazonaws.com), doing this enables more detailed configuration that you’ll need for website hosting.

Origin Settings

For the Default Cache Behaviour Settings you could pretty much leave it as default, except perhaps change it to Redirect HTTP to HTTPS. It’s easy enough to come back in and tweak these settings at any point in the future so don’t dwell on the specific options.

Default Settings

Continuing down, Price Class, I chose Use only US, Europe and Asia but you may choose to vary this (if you’re huge in Brazil perhaps) :)

Alternate Domain Names (CNAMES), here you want to put your desirable domain names that your site visitors will type in, so for easyas.info it’ll be both easyas.info and www.easyas.info.

SSL Certificate, for now just leave it as the Default CloudFront Certificate, we’ll revisit this in a little while.

Finish Create Distribution

So that’s basically it for CloudFront, click the Create Distribution button to finish. Take note of the Domain Name for the Distribution in the list to use in the next step. The distribution creation will likely take up to 15 minutes.

Origin Settings

Configure a domain name [Route53]

So up to this point we’ve created our bucket and configured the CloudFront distribution but we need to actually point those CNAMEs from the previous step at the distribution. For that, we need Route53.

Without a domain name, you’ll have to use the bucket address, similar to this: http://easyas.info.s3-website-us-east-1.amazonaws.com/ or the CloudFront distribution: https://d1u0vlids9dn5r.cloudfront.net, neither of which are exactly memorable :)

Just because we can, returning to our command line, we configure the CNAMEs in Route53.

First, we add a Hosted Zone for our domain: Click the Create Hosted Zone button and simply put in the domain name, i.e. easyas.info.

Create Hosted Zone

This could alternatively be done via command line with something like this:

aws route53 create-hosted-zone --name easyas.info --caller-reference 201606024-2308

After creation you’re taken into the Hosted Zone which allows us to Configure the DNS entries, so hit the Create Record Set button. I’ve used www.easyas.info for the name, create an A - IPv4 Address, with Alias = yes, and the Alias Target set to the Domain Name from the CloudFront Distribution i.e. d1u0vlids9dn5r.cloudfront.net, which you can choose from the dropdown.

Configure www.easyas.info

The click Create.

Create another entry with no www in the name i.e. just easyas.info.

Configure easyas.info

And that’s it! The DNS will take a while to propagate depending on your ISP or other random acts of DNS, but once it does you’ll be able to hit your site with a nice and simple domain.

Adding a real TLS certificate to finish off [ACM]

Whilst the TLS certificate served up by CloudFront is a valid certificate, it will likely prompt some browser warnings (Chrome might even say the world is going to end) as the host names don’t match. What we really need is our own certificate for our domain, with AWS Certificate Manager (ACM) this is both easy and free.

Go back into your CloudFront distribution and edit it. In the SSL Certificate section, choose the Custom SSL Certificate option this time. The hit the Request an ACM certificate button.

This launches a short wizard in a new page. First off, just put in the domain name with and without the www.

ACM Wizard 1 of 3

You could also put in other domains that you might want to use on this certificate. Click the Review and request button.

On the second step, confirm you’ve entered your domain(s) correctly then hit Confirm and request.

ACM Wizard 2 of 3

One the third and final screen you’re informed that a whole pile of email addresses have been sent an email requiring the domain(s) to be validated. Typically, there will be an address that should be accessible by you.

ACM Wizard 3 of 3

Importantly, you need to approve all of these domain(s), so in my case I’ve received two emails (for both www.easyas.info AND easyas.info) and clicked on the link in both of them.

Validation of Domain

In the ACM management screen you’ll now have an issued certificate.

Issued Certificate

This leaves us with just one last step, choosing the certificate in the CloudFront distribution. Find the other browser window for your CloudFront distribution and hit the little refresh button, in the dropdown you’ll now have your new certificate.

Choose your Certificate

Hit the Yes, Edit button at the bottom, and you’re done! The CloudFront distribution will be redeployed again, that’s another 15 minute or so wait, but once done you’ll be able to check your site and see the new certificate in use.

Valid Certificate

So in summary we’ve:

  • made an S3 bucket and put a simple page in there
  • created the CloudFront distribution
  • pointed a domain at the distribution for an incredibly easy way to host a static website, https://easyas.info
  • created a real TLS certificate with ACM to make it all legit.

A note on pricing, the cost of using these services is almost non-existent with the S3 bucket and the CloudFront distribution costing a few cents per 1000s of requests, ACM is free.

Included file 'facebook_like.html' not found in _includes directory