A Drop Box Powered Blog

Dropblog is an effort to automate the majority of my blogging work, using simple software tools. The basic goal is: "create content as normal, leave publication and formatting to software." The blog you are reading (as of December 23, 2014) is powered by this software, and the project is hosted on github for from which others may learn or contribute.

The key goals of the project are:

  • Write content files (documents) in markdown with a minimum of meta data.
  • Store files and images in a standard folder structure, along with other files.
  • Reference images relatively, allow them to be hosted in S3

These goals are achieved by developing the blog as a Ruby on Rails application, hosted on Heroku. The application includes a background job that interfaces with the Dropbox API, Amazon Web Services S3, and the application itself.

Architecture

The application leverages several existing web services to smooth the blogging workflow:

  • Use the Dropbox webhook to notify the application of file changes.
  • Use a Sidekiq worker process to do the heavy lifting of downloading the changed files, publishing pictures to S3, storing image information in the database, and updating blog content in the database.
  • A simple rails application will serve as both the API side and the front facing website which can be modified and published automatically with CI
  • Since only changed files will be re-updated, we won't constantly process pictures.

The application is configured to reference my dropbox account, through Heroku configurations, thus any changes to a specified folder will notify the app that changes have occurred. The app then asks the Dropbox API for a "delta" and depending on what files have changed, runs various processing routines.

In summary, these routines:

  • Create or updated an "article" or "project" markdown file, transforming it into a renderable document.
  • Create or update a file with an image extension, a "picture," uploading it to an S3 bucket, and saving its hosted path along with picture information as a related model to an article or project.
  • Removing files that have been removed from the folder structure, be they pictures or documents.

Folder Structure

In order to provide a simple structure to manage the various files related to both the blog itself, as well as my projects, and related project articles, a standard folder structure is employed. Something like this:

    blog
        projects
            public
                project-one-slug
                    project.md
                    articles
                        article-one-slug.md
                    pictures
                        photo-one.jpg
                project-two-slug
                    project.md
                    articles
                        another-article-one-slug.md
                    pictures
                        another-photo-one.png
        articles
            one-off-article-slug
                article.md
                image1.png
                image2.png
  • Projects are defined by a folder under the projects/public directory. The folder name is the resulting web available slug. For example project-one-slug would be available at http://neverstopbuilding.com/project-one-slug.
  • Under a project folder there can be any related pictures in the pictures folder and articles in the articles folder, these represent project updates, that take blog form, but are specific to the idea of a project.
  • The project information itself (such as this very document) is stored in a project.md file in the root of the project folder. This is designed to capture an overview of the project, or all the project information, should there be no related articles.
  • The name of the project article file becomes the web available slug of that article. All are "pretty," so in this case article-one-slug is available at http://neverstopbuilding.com/article-one-slug.
  • The top level articles folder is similar to the projects folder, only that here, each sub folder represents the slug of the individual article, and that folder contains an article.md containing the article, as well as any related pictures.

The slight difference in handling pictures for projects and articles comes from the though that, one off articles would have specific pictures, but projects may have a library of pictures during their life time, and these might be referenced by multiple articles and the project documentation.

Markdown Formatting

I wanted to have as little meta data as possible in any article, and wanted the markdown files to be able to stand alone when viewed from the computer. So they should begin with a title, and have sub headings, and reference images locally. A simple example might look like:

# My Title
Some content.

## A Subheading
More content.

That's it! This is sufficient to create an article or project document.

Categories and Dates

There are however a few nuances that improve the process, namely the very basic meta data that can be embedded in the title:

# 2014-12-30 How to Make Burgers (cooking)
Some content.

## A Subheading
More content.

The title should stand alone, and I didn't want a cluttered front matter, so the title supports a simple unix date and then a category in parentheses. The whole title is stripped out of the content on render, and this data is parsed out to update the blog. The date sets a hard "created_at" for the article, this was useful for back filling articles from my old blog. The category or "interest" serves as a bare bones way to organize the various documents according to interest, to improve discovery on the website.

Pictures

What's a blog with out some illustrative pictures? The markdown needs only to contain a relative reference to the pictures as organized in the above folder structure:

# My Project Article Title
Some content.

## A Subheading

Look at this image:
![some alt text](../pictures/some-image.png)

When authoring the document, most markdown editors (I use Mou,) will preview the relative images in line. This makes authoring super simple, and gives me a standard place to save any and all pictures, or images related to a project or article. They all just go in the correct pictures folder.

Picture Formatting

Pictures saved to various pictures folders with the extensions jpg, png, gif, jpeg, or svg will be automatically synced to the S3 bucket for the site once they are synced to dropbox. Dropbox does not support using their public links as "hotlines" for blogs and stuff, so I basically replicate an image host with S3. When the files are synced, the public path is saved to the database along with the plain file name. These records are associated with the respective article or project to which the pictures are related.

Durring rendering, when the Markdown parser reads in an image tag, it will look for a relative path or a simple url. If it finds a relative path, then it looks up the related image by file name and renders the full, hosted, public path. This lets us author locally, and then render the picture from a performant image host.

Title Pictures

One of the banes of my blogging existence until recently was the full width title images that I associated with each article. I had to find the image, crop it, save it, upload it to somewhere (often Evernote or Google Plus) then paste the url in the meta data of the post. What a bore.

Now the first picture with the file name of title will be used as this header image for the article or the project. If there is no title picture, then the first picture associated with the document will be used, finally, there is a default header picture as fall back.

A few other header pictures for the static content pages are simply stored as Rails assets, with the whole lot of them synced to S3 as the asset host on deployment.

A Walkthrough

Reading about this project is not as much fun as watching it in action, it really is quite fast, even considering uploading some reasonable images. All of this is managed through free services attached to the Heroku instance, as well as AWS S3, and the Dropbox API. Enjoy:

If you enjoyed reading this or learned something, please consider sharing via , , or . Thanks!

If you're curious about this project, check out the related articles below:

How To Dropblog For Newbies

Last Updated on March 1, 2016 in Software

Like many of you reading, I am new to the world of blogging (at least, I hope I'm not the only newbie here!) Because my Boss, Jason, was tired of... read more

Introducing Dropblog

Last Updated on March 8, 2016 in Software

Welcome to Dropblog. Blogging with "cloud" help! As you can see, there have been a few changes to the site, but the real magic is under the... read more

A full list of projects can be found on the Projects page.