Recently I found myself wanting to move a blog I had over on Wordpress.com to Ghost. I figured it would be a fairly simple process, but as is often the case it turned out a little more involved. Not least because I wanted to avoid having to pay almost $300 to be able to do it...!

Back in 2017 I setup a travel blog to capture some of my adventures with my girlfriend as we moved around the world.

For ease of use I opted for a paid Wordpress plan, even though I was already self-hosting this blog using Ghost at the time. My thinking was that I needed something that was simple enough to not require any special tweaking or setup, and allow me to write on the go using only an iPad. Wordpress had a handy app for that, so sans laptop but coupled with a BlueTooth keyboard, that's what I did.

I was never happy with it though. Wordpress to me is just too bloated for a simple, bog-standard blogging platform. What's more, the basic paid hosting plan is quite limited, not allowing you to change your theme aside from a handful of options, none of which looked very good. Oh, and it was slow.

It may power 30% of the Internet, but I'd rather use something else.

So, when my subscription recently came up for renewal, I figured it was as good a time as any to spend some time migrating it to Ghost, and hosting it in the same way I host this one.

This should be easy I thought, as after all there is an entire guide to do exactly this, using an official Ghost plugin for Wordpress. Great!

Except, when I tried to install the plugin, I was greeted with:

Okay, maybe it's worth it to upgrade just to do this migration. After all once it's done, I'll no longer need a paid plan at all so it'll be worth it in the long run, eventually.

Let's upgrade!

Ouch! Maybe it's just me, but I'm not really down with paying $296 just to move a few blog posts over from one place to another.

There's only a handful of posts, so I could have just recreated them all in Ghost manually, copying over the content, tweaking the formatting, and uploading the images again. But, that's not the developer way, and also, what if I did have hundreds or more to migrate?

Instead I figured there must be a way to do this all for free, so I set about finding it. My goals were:

1. Migrate all of the post content, including images
2. Migrate all of the post comments from the Wordpress built in system to Disqus
3. Drop the date suffix from every permalink. (I had this setting on Wordpress turned on and regretted it). But, ensure the original URLs still work after everything is migrated, too!
4. Make it all SSL enabled when self-hosted (this comes as standard on Wordpress)
5. Do it all for free!

This post is a step-by-step guide of how I did all of the above. As always, if you have any comments, questions, or feedback leave a comment below!

I have split this up into three parts:

• Part 1: How to migrate a paid Wordpress blog to Ghost, completely free! (this post)
• Part 2: Migrate comments from Wordpress to a Ghost blog with Disqus (coming soon)
• Part 3: Setup SSL for a free on a Dockerised Ghost blog with Let's Encrypt (coming soon)

So are you ready? Grab a cup of tea and let's get started!

Steps involved

Let's start with a high-level overview of what we're going to do, so it's clear in our minds as we progress through each step:

1. First, we will setup a Wordpress install locally, and import the content from our paid plan. The open-source version does not have any of those pesky limitations with installing plugins
2. Next, we'll install the Ghost plugin to our local installation. We'll then be able to export an archive of our blog in ghost format
3. Now we can setup a normal Ghost install, and import the archive we just exported
4. At this point, we have our blog content and images moved over, but we still have a little bit of cleanup to do. We'll tweak the CSS and setup redirects so the old permalinks still work. While we're at it, we'll purge any images that we no longer need, to keep the total size down
5. We'll setup Disqus, and migrate over the comments
6. Finally, we will setup SSL using Let's Encrypt

At that point we should have everything replicated on Ghost, and it's just a case of changing the DNS entries to point there instead of Wordpress.

Okay, time to get stuck in!

Setup Wordpress locally in Docker with Docker Compose

Wordpress is actually open-source, and lives on Github here. When someone talks about the self-hosted version, they are referring to https://wordpress.org/. The paid version, by comparison, lives at https://wordpress.com/.

Anyone can use Wordpress for free if they're willing to host it themselves, or you can pay someone else to host it for you. And one of those companies you can pay to do it for you is... Wordpress.com. It was setup by the co-founder of Wordpress Matt Mullenweg, but is pretty much the same as any other hosting service that happens to use the open-source software underneath.

What you get with the paid version is automatic upgrades, hosting, support, built-in backups and things like that. Basically, you simply pay and then create. The downside is that you are limited in customising and tweaking things and have less control, which is only somewhat alleviated by upgrading to one of the more expensive plans.

For more on the difference between self-hosted and paid Wordpress, check out this post: https://www.wpbeginner.com/beginners-guide/self-hosted-wordpress-org-vs-free-wordpress-com-infograph/

So, because Wordpress.org and Wordpress.com use the same software underneath, we can spin up our own local installation on our laptop, and just import our data.

One of the simplest ways to do that is via Docker. There is an official image for Wordpress, which is a great start. Then it just needs to be paired with a suitable mysql database. And of course, there's an image for that too.

If you don't know much about Docker, you can still follow along but it would be worth reading up on the basics. You can install it for your system here, and read more here.

We could bring up these containers manually, but it's a bit easier to use Docker Compose. This will handle bringing up both the blog and database containers and wiring them together.

Create a new directory somewhere, like wordpress_local, and then create a docker-compose.yml file inside which looks like this:

version: '3.3'

services:
   db:
     image: mysql:5.7
     volumes:
       - db_data:/var/lib/mysql
     restart: always
     environment:
       MYSQL_ROOT_PASSWORD: somewordpress
       MYSQL_DATABASE: wordpress
       MYSQL_USER: wordpress
       MYSQL_PASSWORD: wordpress

   wordpress:
     depends_on:
       - db
     image: wordpress:latest
     ports:
       - "8000:80"
     restart: always
     environment:
       WORDPRESS_DB_HOST: db:3306
       WORDPRESS_DB_USER: wordpress
       WORDPRESS_DB_PASSWORD: wordpress
       WORDPRESS_DB_NAME: wordpress
volumes:
    db_data: {}

There's not much clever stuff going on here. We just setup some environment variables so that everything can talk nicely together. The WORDPRESS_DB_USER corresponds to the MYSQL_USER and similarly for the others variables.

We won't worry about the security of the passwords or anything for this, as this setup is a throwaway. Once we've got our data out, we can just destroy the stack

Now open a terminal in the directory you created, and fire up Wordpress locally with:

docker-compose up -d

You'll see it starting up:

Creating network "wordpress_wanderers_default" with the default driver
Creating wordpress_wanderers_db_1 ... done
Creating wordpress_wanderers_wordpress_1 ... done

Not if you hit http://localhost:8000 in your browser, you'll get the Wordpress setup page:

At this point you can just complete the wizard normally. Don't worry too much about the details, as again this is just a throwaway install.

Once you have created a user you should be able to login, and you will be at the Wordpress dashboard, just like on the paid hosting. Now to bring over our blog content!

Exporting our paid Wordpress blog and reimporting locally

In your paid Wordpress dashboard, generate an export of all your content by going to 'Tools -> Export'. You should export both the content archive as well as the media. Save both to the same folder you created earlier.

Back in our local Wordpress dashboard, we can now just head over to 'Tools-> Import', and click to install the importer:

Once done, 'Install Now' will change to 'Run importer', where you can then import the content archive you saved.

The content archive is the single xml file, inside the first zip archive

You'll need to map every user from your paid blog to one on the local one. If like me that's just the same single user, it's pretty straightforward. You can also tick the option to 'Download and import file attachments' which will try to retrieve the images automatically, so we can leave the media archive we downloaded just as a backup if this does not work.

Let that sit tight and import everything. Depending on how much content you have, it could take a little while.

If you get a message for some items that they failed to import, then you may have some custom plugins on your paid install that add different post types etc. In this case you might need to install them locally too/add some tweaks before those items will import. I had this problem with feedback posts, from the JetPack plugin.

Now if you hit http://localhost:8000 again, you should see your posts (the theme is probably ugly, but we've moving it all to ghost anyway so we'll tidy it all up there).

Exporting to Ghost

Time to move everything to Ghost. First, install and activate the plugin. Now we're hosting Wordpress ourselves, we can install anything we want!

Then also install the 'Categories and Tag Converter' under 'Tools -> Import'. We'll need this to map Wordpress categories to Ghost tags.

Once the latter is installed, we can run the tag conversion. Check all, then hit that convert button.

Finally, we can now run the Ghost export plugin and download a zip archive of all of our blog in Ghost format, nice!

Great, so we now have something we can import into a fresh Ghost installation. At this point you could find a paid hosting provider that sets everything up for you, or you could run your own stack as I do using Docker. A walkthrough for that is here, if you're interested.

It doesn't matter how or where your Ghost blog is hosted, so pick whichever makes most sense for you.

Now when you have your new install ready, you can head over to the Labs page and import your archive.

So now we've managed to get all our content migrated over from Wordpress to Ghost, and all for free. So far so good!

Updating the slugs to remove the dates

This step is something you can skip if it's not relevant to you.

When I setup my blog on Wordpress, I had the settings checked to create the slug for each page using the date it was published, like this: https://wanderersoftheworld.com/2018/12/10/24-hours-on-a-bus-journey-to-vietnam/.

In my mind it makes content look dated, is cluttered, and possibly could affect search engine ranking and traffic too.

We can change it, but what about any references kicking around on the Internet or that people have bookmarked, how do we ensure those will still work? Fortunately it is very easy to do this using Ghost. We can upload a URL redirect map, which says for a given from url what it should be mapped to. And even better, it supports regular expressions. This means that we can:

1. Remove the date from the Post URL section of each posts settings
2. Add a redirect from the old link to the new one, so that anyone (or any search engine) that uses it will still find your post

The mapping we need is as simple as a single redirects.json file you can create:

[{"from":"^/\\d{4}/\\d{2}/\\d{2}/(.*)","to":"/$1","permanent":true}]

If your date format is slightly different, you might need to tweak the regex

With this we match any url starting with /yyyy/mm/dd/x and replace it with just x.

All you need to do is upload this file and voila, hitting one of the old links will automatically map to the same without the date. Great!

For more detailed info on redirects, see the Ghost docs on it

What's next? Well you will probably find that while out of the box things look fairly good, you may want to make a few visual tweaks.

Let's do that now!

Tweaking the CSS and styling

The default theme for Ghost is Casper, which is nice and clean and often good enough. This blog uses it too, aside from a few small tweaks I made. One way to make those tweaks is to fork the theme (it's open source after all), after which you can make any modifications you like before packaging it up and importing it back to your blog.

That's great, but is there a quicker, easier way if you just want to make a few, small tweaks here and there? Yes. Via code-injections.

Ghost includes out of the box the ability to add custom CSS/JavaScript code to both the entire blog and even to individual pages. This can be incredibly powerful!

To do this site-wide, just go to 'Settings -> Code Injection'. Now you can add arbitrary code to the header or footer on each page. Below are some examples of things you could easily change.

Add any custom CSS between <style> and </style> tags, as shown in the first example

Add beautiful, free fonts

Found a nice font over at Google Fonts? You could add it and use it with something like:

<style>
@import url('https://fonts.googleapis.com/css?family=Amatic+SC&display=swap');

/* Make title a nicer font (and bigger) */  
.site-title {
    font-family: 'Amatic SC', cursive;
    font-size: 8rem;
}
</style>

Remove the social media and TryGhost links

If you would rather not have them (like me), just hide them:

/* Remove social links in top-right */    
.social-links, .floating-header-share {
  display: none;
}

.site-footer-nav a[href="https://twitter.com/tryghost"] {
  display: none;
}

Change the colour of the progress bar

/* Override the default colour of the progress bar */      
progress::-webkit-progress-value { background-color: #98b4de !important; }
progress::-moz-progress-bar {background-color: #98b4de !important;}
progress {color: #98b4de;}

Add an Instagram link at the bottom

This one requires a bit of custom JavaScript, best placed in the footer:

<script>
// Remove social-links for facebook and twitter, replace with instagram
$(".site-footer-nav").empty()
$(".site-footer-nav").append('<a href="http://instagram.com/yourusername" target="_blank" rel="noopener">Instagram</a>')
</script>

Change the position of an image on an individual blog post

Maybe after doing the import, you find that just a handful of the cover images across your blogs look really bad. Maybe they're too big, or showing too high up or something similar. Well instead of redoing all your images, you could selectively apply a little custom CSS to 'fix' just these. To do that just go to 'Settings -> Code Injection' from the editor window for a single post.

Then you can make the micro-adjustments you need:

<style>
@media (min-width: 800px) {    
    .post-full-image img {
        height: 500px;
        object-position: 0px -600px;
    }
}
</style>

Hopefully that gives you an idea of the kind of things you can do. If you want to make more rigorous changes then either forking the theme or finding another one you like might be better. But for simple quick things like this, it works great!

Purging unused images

You might find you have lots of images that are not referenced and just sitting unused, taking up space. Wordpress tends to create different versions of the same one which can lead to a bit of clutter.

There's a handy open-source tool, ghost-purge-images, which can help with this.

If you've setup your Ghost blog in docker, we can get this working without much effort. We'll just exec into our container and add the package, via npm.

docker exec -it <yourblogcontainer> bash
npm install -g ghost-purge-images

Now it's installed, we need some API keys for it to work. For that, just open the Ghost admin panel and go to 'Integrations' and add a 'Custom Integration'.

Click Create, and you'll get a Content API Key and Admin API Key.

Now back in the blog container shell, run:

cd /var/lib/ghost
ghost-purge-images display --content-key=YOUR_CONTENT_KEY --admin-key=YOUR_ADMIN_KEY

If your setup is different, you basically want to run the command from the directory where your config.development.json or config.production.json file is. Also for this to work, you need to have the url setting in your config file pointing to where Ghost is running. If you're using my Docker setup, you likely don't have that as it is set by environment variables instead. You can add it though with ghost config url http://localhost:2368

If all is well, you'll be given a summary of images that are not in-use. You can then run the same command but changing display to purge to actually remove the files.

👇 Unused images that can be removed:
- content/images/2019/12/DSC00011.jpeg (0.08 MB)
- content/images/2019/12/DSC00011_o.jpeg (0.08 MB)
- content/images/2019/12/favicon.ico (0.10 MB)
- content/images/size/w1000/2019/12/DSC00011.jpeg (0.08 MB)
...
...
- content/images/size/w600/2019/12/DSC00011.JPG (0.04 MB)

📊 Summary:
- 18 files of 353 uploaded images (5.10%)
- Total space: 8.15MB

❔ Want to delete this files? Run `ghost-purge-images purge --content_key=YOUR_CONTENT_KEY --admin_key=YOUR_ADMIN_KEY`
🎁 Open source tool by https://ghostboard.io

Make sure to take a backup of your images first, just in case!

Okay that's it for this one. We've now migrated our blog over to Ghost and done some customisations, all for free. At this point you could update the DNS records for to point to your shiny new blog.

But, there's a little more we may still want to do. So, in future posts, we'll look at look at how to migrate the comments over to Disqus and how to get SSL setup with Let's Encrypt.

Until then have fun customising your new blog over in the Ghost world!

Do you have loads of GoPro movies eating up disk space? Looking for a way to compress them, but in such a way that quality is still very high and they continue to play nice with GoPro software, like Quik? I did too.

I've owned a GoPro Hero 3+ and now more recently a Hero 5. With the right conditions and settings it takes some really nice video, but sometimes they're just a little bit too space hungry for my liking.

Whether you are uploading videos to the cloud, streaming them via a home NAS, or just short on drive space, you might find yourself wanting to reduce the size of your video files.

I've previously written about how to compress videos generally, but for GoPro clips we need to do a little more. That's because there are additional streams embedded in the videos, as well as proprietary metadata.

Here I will walkthrough the process of compressing a video shot from my Hero 5, using FFmpeg. If you just want to skip straight to the final command, scroll to the end.

If you have a different model GoPro, some of the metadata might be slightly different (earlier GoPro's do not have a GPS sensor for example). Hopefully there is enough information here so that you can tweak as necessary for your own camera

Here's our test video:

This video was shot at 2.7k to capture the amazing landscapes in New Zealand. I'd like to retain a decent quality, but reduce the size from 78.9MB.

If you haven't already, install FFmpeg. Now based on my previous post, we could start off by re-encoding just the video at a slightly higher crf factor, like this:

ffmpeg -i GOPR5687.MP4 -c:a copy -c:v h264 -crf 22 output.mp4

Running this spits out a 30.4MB file, around 61.5% smaller, and the quality is fine for my purposes. Not bad!

The range of the CRF scale is 0–51, where 0 is lossless, 23 is the default, and 51 is worst quality possible. A lower value generally leads to higher quality, and a subjectively sane range is 17–28. Consider 17 or 18 to be visually lossless or nearly so; it should look the same or nearly the same as the input but it isn't technically lossless. You should experiment with different crf values to find the sweet-spot for you.

Is that it? Are we done? Not quite.

Keeping GoPro metadata

If we try to import our new smaller video into Quik so that we can do some basic editing, we hit a roadblock.

Hmm, it seems something has happened to the video during the conversion, and now GoPro's software cannot use it.

Let's investigate.

We can use FFmpeg with no options specified to sneak a peak at the metadata in the videos without actually doing anything to them, and compare the original with our new, smaller file.

Here's the original:

~/Desktop  ffmpeg -i GOPR5687.MP4
ffmpeg version 4.0.2 Copyright (c) 2000-2018 the FFmpeg developers
...
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'GOPR5687.MP4':
  Metadata:
    major_brand     : mp41
    minor_version   : 538120216
    compatible_brands: mp41
    creation_time   : 2017-08-07T14:43:26.000000Z
    location        : -43.7941+170.1170/
    location-eng    : -43.7941+170.1170/
    firmware        : HD5.02.01.57.00
  Duration: 00:00:10.41, start: 0.000000, bitrate: 60633 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709), 2704x1520 [SAR 1:1 DAR 169:95], 60541 kb/s, 59.94 fps, 59.94 tbr, 60k tbn, 119.88 tbc (default)
    Metadata:
      creation_time   : 2017-08-07T14:43:26.000000Z
      handler_name    : 	GoPro AVC
      encoder         : GoPro AVC encoder
      timecode        : 14:58:24:55
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      creation_time   : 2017-08-07T14:43:26.000000Z
      handler_name    : 	GoPro AAC
      timecode        : 14:58:24:55
    Stream #0:2(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2017-08-07T14:43:26.000000Z
      handler_name    : 	GoPro TCD
      timecode        : 14:58:24:55
    Stream #0:3(eng): Data: none (gpmd / 0x646D7067), 33 kb/s (default)
    Metadata:
      creation_time   : 2017-08-07T14:43:26.000000Z
      handler_name    : 	GoPro MET
    Stream #0:4(eng): Data: none (fdsc / 0x63736466), 14 kb/s (default)
    Metadata:
      creation_time   : 2017-08-07T14:43:26.000000Z
      handler_name    : 	GoPro SOS

And here's our smaller conversion:

~/Desktop  ffmpeg -i output.mp4
ffmpeg version 4.0.2 Copyright (c) 2000-2018 the FFmpeg developers
  ...
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.12.100
    location-eng    : -43.7941+170.1170/
    location        : -43.7941+170.1170/
  Duration: 00:00:10.41, start: 0.000000, bitrate: 23356 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc), 2704x1520 [SAR 1:1 DAR 169:95], 23252 kb/s, 59.94 fps, 59.94 tbr, 60k tbn, 119.88 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      timecode        : 14:58:24:55
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
    Stream #0:2(eng): Data: none (tmcd / 0x64636D74), 0 kb/s
    Metadata:
      handler_name    : TimeCodeHandler
      timecode        : 14:58:24:55

Straight away we can see that the output from the original file is considerably longer than the one for our converted file.

There are two main differences between the two:

1. The original file contains 5 separate streams embedded inside it, our transcode just 2
2. The original file uses proprietary handler_name's like GoPro AVC, whereas our transcode uses generic versions, like VideoHandler

What happened is that during our transcoding we essentially stripped out loads of data which the GoPro camera writes into the file, meaning we've lost some information as well as the ability to use our files in GoPro software.

Not ideal, let's fix that.

Multiple Streams

As you would expect, a video is generally made up of both an audio and a video part, and each part is referred to as a stream.

What might not be so intuitive at first though is that a file can, and often does, include more streams than just one audio and one video. For example, we might have several different audio streams for different spoken languages, plus a subtitle stream. A decent video player will allow us to choose which streams we want to use when we play the file, perhaps french audio with english subtitles.

In the case of our GoPro, we have several additional data streams, one of which contains the GPS data that is recorded if it is switched on (seen above as GoPro MET).

By default, FFmpeg includes just one audio and one video stream in the output file, choosing what it considers to be the best of each type.

To tell FFmpeg to copy all streams we can use -map 0. Also, to tell it to copy streams even if it doesn't recognise their content (necessary for some GoPro data streams), we can use -copy_unknown also.

ffmpeg -i GOPR5687.MP4 -map 0 -copy_unknown -map_metadata 0 \
    -c copy -c:v h264 -crf 22 output.mp4

Since we want to preserve most of the streams without modification, we pass -c copy first, which says to just copy each stream in the input directly to the output. Then, we override the copy codec just for the video stream, using -c:v h264 as before.

I've also added -map_metadata 0 to copy all of the global metadata from input to output.

If we inspect the video again, it's better. We have now 5 streams as we wanted:

 ~/Desktop  ffmpeg -i output.mp4
ffmpeg version 4.0.2 Copyright (c) 2000-2018 the FFmpeg developers
...
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    creation_time   : 2017-08-07T14:43:26.000000Z
    encoder         : Lavf58.12.100
    location-eng    : -43.7941+170.1170/
    location        : -43.7941+170.1170/
  Duration: 00:00:10.41, start: 0.000000, bitrate: 23407 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc), 2704x1520 [SAR 1:1 DAR 169:95], 23252 kb/s, 59.94 fps, 59.94 tbr, 60k tbn, 119.88 tbc (default)
    Metadata:
      creation_time   : 2017-08-07T14:43:26.000000Z
      handler_name    : VideoHandler
      timecode        : 14:58:24:55
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      creation_time   : 2017-08-07T14:43:26.000000Z
      handler_name    : SoundHandler
    Stream #0:2(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2017-08-07T14:43:26.000000Z
      handler_name    : TimeCodeHandler
      timecode        : 14:58:24:55
    Stream #0:3(eng): Data: none (gpmd / 0x646D7067), 32 kb/s (default)
    Metadata:
      creation_time   : 2017-08-07T14:43:26.000000Z
      handler_name    : GoPro MET
    Stream #0:4(eng): Data: none (stts / 0x73747473), 14 kb/s (default)
    Metadata:
      creation_time   : 2017-08-07T14:43:26.000000Z
      handler_name    : DataHandler

We still have two issues though!

1. The video clip will still not open in Quik... probably has something to do with those generic handler names
2. While we were transcoding, ffmpeg spat out a warning for the SOS stream:

[mp4 @ 0x7ff4cc006c00] Unknown hldr_type for fdsc, writing dummy values3.0kbits/s speed=0.148x

Let's continue and try to fix those now.

Correcting Handler Names

Like most things, it is possible to customise the handler name with FFmpeg. We just need to specify the handler metadata tag for each stream and give the correct name.

For example, -metadata:s:v: handler='  GoPro AVC' sets the handler_name for the video stream to be called GoPro AVC.

Although the written metadata is called handler_name, the tag to set it is just handler. This has caused confusion for at least some others

We can now try to transcode the video, making sure to set the names for the audio and video tracks:

ffmpeg -i GOPR5687.MP4 -copy_unknown -map_metadata 0 \
-c copy -c:v h264 -crf 22 =\
-map 0 \
-metadata:s:v: handler='  GoPro AVC' \
-metadata:s:a: handler='  GoPro AAC' \
output.mp4

This time, all is well if we try to import it into Quik.

Great, so we can override the handler names to the GoPro specific ones! One slight snag though, the streams are not always in the same order. That means if we set the handler names by index, for some videos we might end up using a name of GoPro AAC for the video, when that's the name for the audio.

This might not be a problem if you're just encoding a single file, as you can check it first to see what order it has been recorded in. But, if you want to batch convert many with the same command in a script for example, it will bite you.

Selecting streams by name

Fortunately, we can work around this by explicitly listing the input streams that we want to map, and then follow that with the handler names we would like for each in the same order. This works because FFmpeg will map the output streams in the same order as you list them from the input.

We can explicitly list the video and audio streams easily, as there is only one of them. This just requires -map 0:v or -map 0:a respectively, which says to map the video or audio stream from the 0th (first) input file.

The data tracks are a bit more complicated, as there are 3 of them and they may appear in any order. Luckily, we can also pick the input streams by name:

-map 0:m:handler_name:'  GoPro MET'

Here we specify that we want to map the stream which has a handler_name of GoPro MET in its metadata, from the 0th input file.

Perfect, now we have all the pieces we need!

Putting it all together

So, to compress the GoPro video named GOPR5687.MP4 to a smaller size, but still have it keep it's metadata and work in Quik, we can use something like:

ffmpeg -i GOPR5687.MP4 -copy_unknown -map_metadata 0 \
-c copy -c:v h264 -crf 22 -pix_fmt yuvj420p \
-map 0:v -map 0:a \
-map 0:m:handler_name:' GoPro TCD' \
-map 0:m:handler_name:' GoPro MET' \
-map 0:m:handler_name:' GoPro SOS' \
-tag:d:1 'gpmd' -tag:d:2 'gpmd' \
-metadata:s:v: handler='        GoPro AVC' \
-metadata:s:a: handler='        GoPro AAC' \
-metadata:s:d:0 handler='       GoPro TCD' \
-metadata:s:d:1 handler='       GoPro MET' \
-metadata:s:d:2 handler='       GoPro SOS (original fdsc stream)' \
output.mp4 \
&& touch -r GOPR5687.MP4 output.mp4

Here with the -map lines we extract, in order, the video, the audio, and then the TCD, MET and SOS data streams. Then in the -metadata lines we name each stream accordingly.

Watch out! What looks like a space before the handler_name's is actually a TAB character. Yes, GoPro really does record handler names starting with a tab. And yes, for some streams like MET, if it doesn't match exactly, Quik won't recognise it. Spent some time tearing my hair out before I realised this...

Remember the warning about dummy data being used for the SOS fdsc stream I mentioned earlier?

That happens because FFmpeg doesn't know what an fdsc stream is, so strangely instead of just copying it anyway (which is what we asked), it decides to stuff the whole thing with garbage data. It seems like this SOS stream is only used for file recovery anyway, and isn't that important.

Nevertheless, I'd still prefer to copy it over if possible. To do this, I've added a small hack which retags the 'fdsc' stream as 'gpmd' using -tag:d:2 'gpmd'. Because FFmpeg is familiar with this type, it will happily copy across the data. Then, when I rename the handler, I've given it a name to indicate that it was originally the fdsc stream.

If you don't care about keeping this stream, you can omit these tags. Hopefully in the future FFmpeg will just copy all the data if -copy_unknown is specified anyway, so that there is no need for it anyway

I've explicitly specified the output pixel format as -pix_fmt yuvj420p also, so there is no guesswork on behalf of the codec.

If you have any issues with colour reproduction, you might also want to look at colour profile settings. See here or here for a little more info.

Finally, I've added atouch command at the end, in order to copy across the original file modification timestamps to the newly created file. Handy if we're going to be sorting the files by date!

So now if we run the above command, we generate a much smaller video that retains our metadata, and keeps Quik happy:

In my tests, the GPS guage data is copied over and you can activate the guages within Quik to overlay onto your video. However, the data doesn't seem to match up correctly, despite all being there (and I can extract the gps coordinates recorded without issue). I suspect there is some timing data that is not copied exactly. There is a reddit thread that is relevant to this. There has also been commits to the FFmpeg codebase from a GoPro engineer to support the gpmd stream, so perhaps in time this will work correctly.

That's all there is to it! And if you want to automate this process for a number of videos, I created a tool called Shrinkwrap to do this. If you're familiar with Docker you might want to check it out, using the GoPro5 preset.

If you have any comments or improvements, feel free to leave them below!

Post cover image source

Do you have loads of videos littering your drive from your phone, camera, GoPro etc. taking up loads of space? So did I, so I started looking for a way to reduce the size while keeping the perceived quality the same, and retaining all of the original metadata and timestamps.

Storage space might have become a lot cheaper in recent years, but at the same time we're recording more and more high quality video. Also, when you start thinking about backing up all those precious memories to a cloud service then size really starts to matter again.

It's not just space either, over the years I've owned a range of different devices, all recording video in different formats, codecs and qualities. Some of these are now old and difficult to work with, with no native support in macOS, Plex, etc.

Here's just some of the digital cruft that's accumulated for me:

• Home movies from an old digital camera (mpeg2 in a .MOD container)
• Old edited movie projects (mpeg2 in a .wmv container)
• Clips from a very old mobile phone (H.263 QCIF in a .3gp container)
• Video from a more modern Android phone (h.264 1080p in a .mp4 container)
• GoPro Hero 3+ footage (h.264 720p in a .mp4 container)
• GoPro Hero 5 footage (h.264 1080p/2.7k in a .mp4 container)

Perhaps you have something similar if you've used many different devices over the years too!

A 19s clip from a really old phone weighs in at 198kb. On the other hand, 10 seconds of 2.7k on the GoPro5 puts out 78.2mb. That means my GoPro (not at max quality) eats up around 750x more space per second... we've come a long way!

Sure, VLC will play pretty much everything you can throw at it, but it's not always convenient. I'd like to be able to stream my media across my devices, whether in or out the house.

Goals

So, I set about organising my video library, with a few goals:

1. Transcode all old audio/video files to the standards of 2018 (namely h.264 video, and aac audio)
2. Retain as much metadata as possible, in particular creation date and file modification timestamps (so sorting files by date is not messed up, for example)
3. Reduce the file size! (But, retain the same perceived quality)

Transcoding is a lossy operation that re-encodes the entire data stream and repackages it, so there is some loss from the original. However, with the right settings, the difference is almost impossible to notice.

Let's get shrinking

If you google how to reduce video size you'll get a whole range of different results. It's a bit of a minefield, with many blogs and articles set up promoting all sorts of shareware tools all claiming to be your one-stop solution.

There is no need to cough up any money though, because a very advanced and capable open-source tool exists: HandBrake. This handy app is completely free and supports macOS, Windows, and Linux.

HandBrake might look daunting at first, but for most of the options the defaults they've chosen are sensible. If you load a particular preset (iPhone, YouTube etc) then it pretty much just works, and has been able to handle everything I've thrown at it.

Apart from one snag. Handbrake is not the best at preserving the metadata of the original file. That's a bit of a dealbreaker for me, as I'm often sorting and organising files by date in Finder. If I use HandBrake, then every video I've taken ends up having a modified timestamp of whatever day I run them through it. There is an open request on their github to improve this, but as of now it's not high up the priority list.

It's also a little clunky working with files in batch mode. I had years worth of videos I wanted to run through it, so anything that can't be easily automated is not ideal.

There is also HandBrakeBatch, a small wrapper around HandBrake that was made before there was any built-in batch support. It's a very simple tool, but does manage to preserve the timestamps. However, it is no longer maintained and hasn't been updated since 2013.

HandBrake is a very useful tool and if you don't care about preserving all the metadata then you might find it does everything you need, so give it a try.

As preserving the metadata was important to me, I needed something else.

Enter FFmpeg

FFmpeg is "a complete, cross-platform solution to record, convert and stream audio and video." It's a very advanced and powerful tool that can do much more than simple video-transcoding. You can pretty much do anything you can think of to a video.

Let's say we have an old home video shot with a camcorder that was saved as a .MOD and we want to convert it to something more modern.

First, we need to install FFmpeg. Next, we just open up a terminal window (or cmd prompt on Windows), and fire off:

ffmpeg -i input.MOD output.mp4

That's it. FFmpeg recognises the file extensions and uses suitable codecs and defaults for each, so in this case it will take our old input.MOD file and transcode it to output.mp4, which will be h.264 inside an .mp4 container.

You can also explicitly choose the video and audio codecs to use with -codec:v and -codec:a respectively

What about file size?

In my simple test I took an old video clip from a JVC Everio Camcorder, shot in 2010.

The original file was 27.5mb, for a 24 second clip. The transcoded file is 3.4mb, a reduction of ~87%!

Surely we're going to be getting a terrible conversion to be able to make it that small?! Actually no, to my eyes the two are, for all intents and purposes, the same.

Quality

What makes a video seem to have high quality?

One of the key factors that influences this is the bitrate, how many bits of information are used for encoding each second of video. If we have more bits, we can encode more information. Similarly, if we have less bits available to use, then we have to be selective in deciding what information to keep, and what we have to throw away.

There is always a trade off between quality and file-size. Generally speaking, higher quality uses more space.

At the extreme end, a single minute of UHD 4k footage might take up over 5GB of disk space.

The job of a codec is to stuff as much information as possible about your video into the smallest package it can. Overtime, codecs improve and better ways of compressing video are designed that might be able to achieve both higher quality as well as lower file-size.

h.265 is the successor to h.264 which boasts even more impressive compression, sometimes giving as much as a 50% reduction in file-size. The cost is limited support, and the need for fast, modern hardware to make use of it. The tradeoff wasn't worth it for me right now, but given time it will likely take over.

The amount of bits we might want to use is also not necessarily the same throughout all points of our video. For example, if the camera is positioned steady and not much is changing,  then there's not as much to encode and we might get away with using less bits.

On the other hand if we have lots of changes frame to frame, we're going to need more bits to encode it all. But to make it even more interesting, when things are moving the human eye cannot perceive as much detail as when they're static, so for fast-motion content we might also get away with less bits.

Fortunately we don't really have to worry about all this, we can just use the crf factor setting (Constant Rate Factor) from the h.264 codec. And in fact, we've already used it without knowing.

The crf factor basically translates as "try to keep this quality overall", and will use more or less bits at different parts of the video, depending on the content. (the bitrate is variable).

As best described by the docs:

The range of the CRF scale is 0–51, where 0 is lossless, 23 is the default, and 51 is worst quality possible. A lower value generally leads to higher quality, and a subjectively sane range is 17–28. Consider 17 or 18 to be visually lossless or nearly so; it should look the same or nearly the same as the input but it isn't technically lossless.

The range is exponential, so increasing the CRF value +6 results in roughly half the bitrate / file size, and vice-versa.

Let's try this out:

Here's another video shot while skiing, on an old Android phone in 2012.

Let's see what happens if we go crazy and try using a value of 51 for the crf:

ffmpeg -i VID_20120116_121220.mp4 -crf 51 output.mp4

As you can see, although we achieved a drastic reduction in file-size, we had to throw away a huge amount of detail to get there; the video is terribly blocky.

So, the trick with the crf is to experiment with different values for your own videos. Depending on how much you can see the difference, how you're going to watch them, what the original source was and so on, you might choose different values to me.

For my phone videos, from a Nexus 5, I'll typically get a space saving of 60%+ using a crf of 22 (where the difference is not noticeable to me). If the scene is mostly black, for example a video of fireworks or lightning storms, I've seen it be nearer 95%. My guess is that older hardware isn't able to do as good an on-the-fly encoding because of resource limitations, so the space saving can be great for these.

Starting with the default (23) makes sense, moving nearer 18 if you value quality more, and towards 26-28 if you value space savings more.

Transcoding speed

This is all well and good, but how long does it take to run? After all if we have hundreds of files to transcode, we don't want to leave our poor laptop working for weeks!

FFmpeg has a number of speed presets, which change how quickly the transcode will run. The default is 'medium', but you can choose from 'ultrafast' to 'veryslow'.

A slower preset will take longer to run (sometimes significantly longer), and put more demands on your hardware, but it might be able to do a better job and hit the same quality with a smaller file-size.

How come? Imagine you're packing your car boot to go on holiday and you're in a hurry. You're standing at the car and your partner brings you each suitcase, bag or box to put in one by one. As soon as you take each item to pack, you find a space in the car and put it in. At some point the boot fills up, so you start putting things on the seats and on the roof-rack. All packed!

Now imagine that you have to pack the same car with the same items, but you're in no hurry this time. So you lay out all the items on the ground and take your time thinking through what best fits where. Sometimes you'll take something out and rearrange it if you find something else later that better fits the space. You might be able to pack all the same items into the same car, but fit everything into just the boot leaving the seats and roof free.

This is sort of how better compression can work. With more time, the codec can try different things, go over things multiple times, and generally just make a better choice as to what to put where.

I find that medium and fast are good sweet-spots for me on my laptop:

ffmpeg -i VID_20120116_121220.mp4 -crf 22 -preset fast output.mp4

Preserving metadata

We've now managed to compress our videos down to a much smaller size and retain enough quality that we can't tell the different. Great!

Only, so far we've lost most of the metadata doing so.

Not so great.

The original skiing clip has, among other metadata, all of the correct timestamps:

File Modification Date/Time     : 2012:01:16 11:13:54+00:00
File Access Date/Time           : 2018:11:08 16:34:25+00:00
File Inode Change Date/Time     : 2018:11:07 22:20:37+00:00
Create Date                     : 2012:01:16 11:13:54
Modify Date                     : 2012:01:16 11:13:54
Track Create Date               : 2012:01:16 11:13:54
Track Modify Date               : 2012:01:16 11:13:54
Media Create Date               : 2012:01:16 11:13:54
Media Modify Date               : 2012:01:16 11:13:54

In contrast, our new smaller one has lost it all:

File Modification Date/Time     : 2018:11:08 15:16:28+00:00
File Access Date/Time           : 2018:11:08 16:34:01+00:00
File Inode Change Date/Time     : 2018:11:08 15:16:28+00:00
Create Date                     : 0000:00:00 00:00:00
Modify Date                     : 0000:00:00 00:00:00
Track Create Date               : 0000:00:00 00:00:00
Track Modify Date               : 0000:00:00 00:00:00
Media Create Date               : 0000:00:00 00:00:00
Media Modify Date               : 0000:00:00 00:00:00

By default, FFmpeg won't preserve the metadata from the original streams, but we can tell it to with the -map_metadata option:

ffmpeg -i VID_20120116_121220.mp4 -crf 22 -map_metadata 0 \ 
    -preset fast output.mp4

This will copy all the metadata from the first input file (numbered starting from zero) to the output.

We only have a single input, but it's possible to have more when you're combining videos, overlays etc

Let's look at the metadata again now:

File Modification Date/Time     : 2018:11:08 16:48:34+00:00
File Access Date/Time           : 2018:11:08 16:48:35+00:00
File Inode Change Date/Time     : 2018:11:08 16:48:34+00:00
Create Date                     : 2012:01:16 11:13:54
Modify Date                     : 2012:01:16 11:13:54
Track Create Date               : 2012:01:16 11:13:54
Track Modify Date               : 2012:01:16 11:13:54
Media Create Date               : 2012:01:16 11:13:54
Media Modify Date               : 2012:01:16 11:13:54

Better, but we still don't have the file modification time set correctly.

Let's fix that now!

Recovering file modification timestamps

FFmpeg isn't able to copy the file modification timestamp because it is not part of the metadata inside the file, it is metadata of the actual file itself as written by the OS.

Instead, we can use exiftool by Phil Harvey for this. This is a very powerful exif/metadata tool that is primarily used for photos, but has some support for videos too.

exiftool -tagsFromFile VID_20120116_121220.mp4 -extractEmbedded \ 
   -all:all -FileModifyDate -overwrite_original output.mp4

This will extract all the metadata (-all:all) from the original file, and copy it to output.mp4. In particular, we make sure to include the -FileModifyDate from the outside also.

You could also use touch, like this touch -r VID_20120116_121220.mp4 output.mp4 to copy across the modification date. I'm using exiftool though, just in case there is any other metadata that FFmpeg misses out

Now, we have restored the correct file modification time:

File Modification Date/Time     : 2012:01:16 11:13:54+00:00

Awesome!

Only, it's a bit manual and tedious to do this for our entire video collection...

Automating it all with Shrinkwrap

For each video, our flow is the same:

1. Use FFmpeg to transcode and compress the video
2. Recover the file-level metadata with Exiftool

It would be nice to have a tool that bundles up everything we've seen into one simple package, wouldn't it?

For this reason, I created Shrinkwrap.

Shrinkwrap takes as input one or more video files (or directories), shrinks them all, and then wraps them back up with the original metadata. The end result should be videos that are smaller, all the same type, and as close to the originals as possible. Basically, what our original goals were!

To use Shrinkwrap, you just need to install Docker, and then run something like:

docker run -v /path/to/your/videos:/vids bennetimo/shrinkwrap \
    --input-extension MOD --ffmpeg-opts crf=22,preset=fast /vids

The key parts of this command are:

• /path/to/your/videos/ is where the videos are that you want to convert
• --input-extension is the type of videos you want to process, here .MOD
• --ffmpeg-opts is any arbitrary FFmpeg options you want to use to customise the transcode

That's it, just let it run.

By default, each video will be shrunk into a new file of the same name with the suffix -tc.mp4, so that you can distinguish it from the originals. It will convert all video to h.264, and all audio to aac.

The originals are not modified or touched, so you can try out different options and then only when happy, delete the original if you want

Shrinkwrap will use a slightly more advanced FFmpeg command, a bit more like this:

ffmpeg -i "input.mp4" -copy_unknown -map_metadata 0 -map 0 -codec copy \
    -codec:v libx264 -pix_fmt yuv420p -crf 23 \
    -codec:a libfdk_aac -vbr 4 \
    -preset fast "output.mp4"

Woah! Quite a lot going on there!

This translates as "hey FFmpeg, take my video input.mp4 and transcode it, making sure to copy_unknown streams, map all the streams you find, and map_metadata from my input file. For any video, convert it using libx264, with a pix_fmt of yuv420p and a crf quality of 23. For audio, I want it as aac using the libfdk_aac codec using a vbr of 4. Finally for any other streams (e.g. data), just copy them as is, and do the whole thing fast!"

Here you can really start to see the power of FFmpeg. You might want to specify additional filters too with -vf. e.g. if your video is interlaced, you can use -vf=yadif to de-interlace it

For more customisation, you can check the readme. There are also a couple of Shrinkwrap presets that do a few extra things, specifically for GoPro footage, that you might want to check out if that applies to you.

I've also written a separate post specifically for compressing GoPro video files

Now we have everything we need to shrink our ever growing collections and keep them maintainable!

Shrinkwrap is working for my needs right now, but if you have any comments or suggestions, be sure to leave them below!

Have fun saving space :)

Post cover image source

Back in 2015 I wrote a series of blog posts describing a way to setup a Ghost blog using docker and docker-compose.

Since then a fair amount has changed (technology moves fast!). Ghost has updated from 0.7.x releases through 1.x and is now sitting at 2.x, with a number of changes along the way, most of which is not backwards compatible. Configuration has switched to nconf, the main content path has changed, the database is completely updated, and the theme api has had extensive rework. Of course, the changes bring in a whole load of new features, including a much improved editor and support for richer content.

In the docker world a lot has changed too. Links have been officially deprecated, compose file version 1 is deprecated, data-only containers have been deprecated in favour of docker volumes, and volume support has made its way into compose.

So, it's time to update our blog stack to work with Ghost 2.x and bring everything up to date for 2018.

As before I'll walk through how I've set everything up piece by piece, so you can follow along with your own setup, just replace all references to coderunner.io with your own domain :)  If anything is not clear, or you have any other thoughts (or improvements!), drop me a comment below.

I have split this up into four parts:

• Part 1: Setting up a Dockerised installation of Ghost with MariaDB
• Part 2: Deploying Ghost 2.x on DigitalOcean with Docker Compose (coming soon)
• Part 3: Backing up a Dockerised Ghost blog using ghost-backup (coming soon)
• Part 4: Syncing a local and remote Dockerised Ghost blog (coming soon)

The Goal

What we're shooting for:

• Ability to bring up/down the whole stack with a single command (we'll use Docker Compose for that)
• Let us create content and write our posts on a local environment (e.g. laptop) before syncing it easily with a live host once we're ready
• Front our blog with a reverse proxy, because we will be hosting it on a VPS and may want to have other blogs/apps on the same box
• Easy and automated backups of our blog, to our local machine or a cloud storage service like Dropbox.
• Stay as close as possible to Ghosts recommended stack, and Docker best practices

Ready? Let's get started!

Overview

In this first post we will setup Ghost from their official docker image, backed by a MariaDB container, and fronted by Nginx. To wire it all together, we'll use Docker Compose.

Before getting started, you should have Docker and Docker Compose installed. If you're on Mac or Windows, then they both come bundled together

Why docker? Why MariaDB?

Ghost can be setup with either sqlite3 or Mysql/MariaDB.

We want to have our local environment mimic live as closely as possible, so that we can easily sync between the two. Because of this we will avoid sqlite3 (which is only recommended for development), and back our blog using a fully featured DB in both environments. This is one of the benefits of using Docker, that we can easily package up our entire stack so it runs the same in both places. No more 'well it worked on my local machine!' problems.

I've chosen to use MariaDB, but you can use Mysql also if you prefer, just change the docker image. For our purposes, you should be able to drop in one as a replacement for the other.

Directory Structure

So that it is clear up-front, this is the directory structure we'll be putting together:

.
├── config.base.json
├── docker-compose.yml
└── env
    └── coderunner.dev.env

1 directory, 3 files

We'll build up each one as we go, but you can browse all the code for this part on github  if you want to check something, or to refer back to the complete solution anytime.

You can checkout all the files for this part with git clone -b part1 git@github.com:bennetimo/hello-blog.git

Creating a Docker Volume

First things first, we need a place to store the great blog content we'll be creating!

We could store everything directly on the host and then bind mount the volume into the container, but this makes everything less portable and very host-specific; we would have to worry about paths, and making sure they're correct on whichever host we'll be running on.

In the previous version of this post I used a data only container to get round this. That would still work, but since then Docker Volumes have come a long way as well as now being fully supported by Compose. So, this is now the preferred method of storing data within docker.

So let's kick things off by creating a file, call it docker-compose.yml and put it in a folder on your machine, let's call it hello-blog.

In this file we'll be declaratively listing all of the components that make up our stack, and how they compose together.

Here is the first version of our file:

version: "3.7"

# Data volumes containing all the persistent storage for the blog
volumes:
 data-ghost:
  name: data-ghost
 data-db:
  name: data-db

This is a very simple docker-compose file that just declares two volumes, which will then be created for us if they don't exist. We have one that will hold the MariaDB database, and the other that will hold all of the ghost content.

We're using Compose file version 3.7, which at the time of writing is the latest. Anything without a version number is considered version 1 and legacy, see here for more info.

Now that we've got a volume to store our content, we can configure the database to use it.

Setup our database

There's an officially supported image for MariaDB which makes our lives easy.

All we need to do is add it to our docker-compose.yml, as a new section below the version line:

services:
 # Database container
 mysql:
  image: mariadb:10.3
  container_name: "db"
  restart: always
  env_file: env/coderunner.dev.env
  expose:
   - "3306"
  volumes:
   - "data-db:/var/lib/mysql"

There's a few things going on here, so let's go through it.

We're creating a database container using the mariadb:10.3 image that is available on Docker Hub. It exposes the default 3306 port so that other containers can talk to it, and is set to restart automatically if it should ever die.

The container_name isn't required, but we've added it to override the default name that would otherwise be generated, to just be the simpler db.

The container_name is like the external name that we'll see when we interact with our container using the docker cli. The service name is different, here we've called it mysql, and is the internal name used within the docker network. This name is important, and we'll see why later.

The line "data-db:/var/lib/mysql" tells docker that we would like the /var/lib/mysql directory within the container to actually be stored inside our data-dbvolume.

Any other directories that the MariaDB container uses will still only exist within that container

And finally, we also specified an env_file with our db configuration:

## MariaDB configuration
MYSQL_ROOT_PASSWORD=<YOURDBROOTPASSWORD>
MYSQL_USER=<YOURDBUSER>
MYSQL_PASSWORD=<YOURDBPASSWORD>
MYSQL_DATABASE=ghost

Fill in the blanks for your own blog setup.

Setup Ghost

Next up we need to actually add Ghost, and we have an official image for that too, awesome!

 # Ghost containers
 blog:
  image: ghost:2.2
  container_name: "blog"
  restart: always
  env_file: env/coderunner.dev.env
  volumes:
   - "data-ghost:/var/lib/ghost/content"
   - "./config.base.json:/var/lib/ghost/config.development.json:ro"
   - "./config.base.json:/var/lib/ghost/config.production.json:ro"

As before, we want all user content to live inside the volume we created, so we tell docker to store the ghost content directory that lives at /var/lib/ghost/content inside our data-ghost volume.

The only other thing new here is a couple of lines for setting up our Ghost config files.

Since Ghost 1.0, all config is handled via nconf. This means we can use a config.<env>.json file to configure what settings we need for each environment, and Ghost will load the correct file (matching the Ghost environment) automatically if it's located in the correct place.

The nice thing about using nconf is that every setting can also be specified as an environment variable, which if set will override any values from the config file.

So we can have a base config file with any common settings, and override any environment specific settings with environment variables.

We mount the base file config.base.json as both the development and production config files. Here is the content of that file.

And then in our env/coderunner.dev.env file we add the dev specific settings:

# Ghost configuration
url=http://coderunner.io.develop
database__connection__user=<YOURDBUSER>
database__connection__password=<YOURDBPASSWORD>
NODE_ENV=development

The url value is a Ghost config setting to set the url of our blog. And the NODE_ENV sets the environment that Ghost will start in. For the database details, just make sure they match what you set earlier for MariaDB.

How does the Ghost container talk to the database container?

In older versions of Docker we would use container links to network our ghost and database containers together so that they could talk to each other. This had the side effect of making all environment variables defined in one container available to any container it was linked with. While this meant some setup boiler plate was reduced, it had a number of issues and has now been deprecated.

Instead, all containers are now connected to the same network by default. This means that our ghost container can automatically talk to the MariaDB container using its service name mysql, without us having to do anything extra. If we wanted to use a different name, or have multiple hostnames then we could use network aliases.

Our ghost config file has the database host set to mysql, which is the same as the service name, so nothing more is required for the two to communicate.

At this point we could fire up our blog, but we wouldn't be able to access it from our local machine as we're not exposing the ghost ports. We will go one better than exposing the Ghost port directly, and setup nginx.

Put it all behind nginx

By setting everything up behind an nginx reverse proxy, we can have multiple services (applications, other blogs etc) running on a single box and have nginx handling traffic routing between them. We could set this up manually, but there is already an awesome out-the-box Docker setup in jwilder/nginx-proxy.

Now we're really starting to see the magic and power of Docker. We're building our application by sticking together components like lego bricks! If we need a new piece, we first check Docker Hub to see if a suitable one already exists that we can use.

Let's add nginx-proxy to our docker-compose.yml:

# Reverse Proxy
 nginx-proxy:
  image: jwilder/nginx-proxy:0.7.0
  container_name: "nginx-proxy"
  restart: always
  ports:
   - "80:80"
  volumes:
   - /var/run/docker.sock:/tmp/docker.sock:ro

And that's all we need to create a fully-fledged reverse proxy! Now we just need to tell it the hostname that will map to our blog, by adding a single environment variable to the blog container:

environment:
  - VIRTUAL_HOST=coderunner.io.develop

This simple environment variable is all we need to tell nginx to route any traffic destined for the url coderunner.io.develop on port 80 to be handled by our ghost container.

Great!

Chrome and Firefox now redirect all .dev traffic to https, which is why I now use .develop here instead. Otherwise you'd have to mess around setting up SSL certificates etc on your local machine

Start it up!

In the main blog directory:

docker-compose up

And we're running!

On the very first launch the Ghost container might try to connect to MariaDB before it's finished setting up the database. To avoid it you can start MariaDB separately first with docker-compose up -d mysql, or by using my modified image. See here for more info.

We just need to add this mapping to our hosts file:

127.0.0.1		coderunner.io.develop

So that our local machine knows to route the traffic to our local nginx server.

Or you could use a hosts file manager like Gas Mask for macOS

Now we can fire up a browser and visit http://coderunner.io.develop, and we're greeted with Ghost:

We now have a Ghost blog running, linked to a MariaDB container, and fronted by an Nginx reverse proxy, all running in Docker containers. Nice!

Next up, we need to set it up running live on the Internet so, you know, people can actually read it.