Dropblog is an effort to automate the majority of my blogging work, using simple software tools. The basic goal is: "create content as normal, leave publication and formatting to software." The blog you are reading (as of December 23, 2014) is powered by this software, and the project is hosted on github for from which others may learn or contribute.
The key goals of the project are:
- Write content files (documents) in markdown with a minimum of meta data.
- Store files and images in a standard folder structure, along with other files.
- Reference images relatively, allow them to be hosted in S3
These goals are achieved by developing the blog as a Ruby on Rails application, hosted on Heroku. The application includes a background job that interfaces with the Dropbox API, Amazon Web Services S3, and the application itself.
The application leverages several existing web services to smooth the blogging workflow:
- Use the Dropbox webhook to notify the application of file changes.
- Use a Sidekiq worker process to do the heavy lifting of downloading the changed files, publishing pictures to S3, storing image information in the database, and updating blog content in the database.
- A simple rails application will serve as both the API side and the front facing website which can be modified and published automatically with CI
- Since only changed files will be re-updated, we won't constantly process pictures.
The application is configured to reference my dropbox account, through Heroku configurations, thus any changes to a specified folder will notify the app that changes have occurred. The app then asks the Dropbox API for a "delta" and depending on what files have changed, runs various processing routines.
In summary, these routines:
- Create or updated an "article" or "project" markdown file, transforming it into a renderable document.
- Create or update a file with an image extension, a "picture," uploading it to an S3 bucket, and saving its hosted path along with picture information as a related model to an article or project.
- Removing files that have been removed from the folder structure, be they pictures or documents.
In order to provide a simple structure to manage the various files related to both the blog itself, as well as my projects, and related project articles, a standard folder structure is employed. Something like this:
blog projects public project-one-slug project.md articles article-one-slug.md pictures photo-one.jpg project-two-slug project.md articles another-article-one-slug.md pictures another-photo-one.png articles one-off-article-slug article.md image1.png image2.png
- Projects are defined by a folder under the
projects/publicdirectory. The folder name is the resulting web available slug. For example
project-one-slugwould be available at
- Under a project folder there can be any related pictures in the
picturesfolder and articles in the
articlesfolder, these represent project updates, that take blog form, but are specific to the idea of a project.
- The project information itself (such as this very document) is stored in a
project.mdfile in the root of the project folder. This is designed to capture an overview of the project, or all the project information, should there be no related articles.
- The name of the project article file becomes the web available slug of that article. All are "pretty," so in this case
article-one-slugis available at
- The top level
articlesfolder is similar to the projects folder, only that here, each sub folder represents the slug of the individual article, and that folder contains an
article.mdcontaining the article, as well as any related pictures.
The slight difference in handling pictures for projects and articles comes from the though that, one off articles would have specific pictures, but projects may have a library of pictures during their life time, and these might be referenced by multiple articles and the project documentation.
I wanted to have as little meta data as possible in any article, and wanted the markdown files to be able to stand alone when viewed from the computer. So they should begin with a title, and have sub headings, and reference images locally. A simple example might look like:
# My Title Some content. ## A Subheading More content.
That's it! This is sufficient to create an article or project document.
Categories and Dates
There are however a few nuances that improve the process, namely the very basic meta data that can be embedded in the title:
# 2014-12-30 How to Make Burgers (cooking) Some content. ## A Subheading More content.
The title should stand alone, and I didn't want a cluttered front matter, so the title supports a simple unix date and then a category in parentheses. The whole title is stripped out of the content on render, and this data is parsed out to update the blog. The date sets a hard "created_at" for the article, this was useful for back filling articles from my old blog. The category or "interest" serves as a bare bones way to organize the various documents according to interest, to improve discovery on the website.
What's a blog with out some illustrative pictures? The markdown needs only to contain a relative reference to the pictures as organized in the above folder structure:
# My Project Article Title Some content. ## A Subheading Look at this image: ![some alt text](../pictures/some-image.png)
When authoring the document, most markdown editors (I use Mou,) will preview the relative images in line. This makes authoring super simple, and gives me a standard place to save any and all pictures, or images related to a project or article. They all just go in the correct
Pictures saved to various
pictures folders with the extensions jpg, png, gif, jpeg, or svg will be automatically synced to the S3 bucket for the site once they are synced to dropbox. Dropbox does not support using their public links as "hotlines" for blogs and stuff, so I basically replicate an image host with S3. When the files are synced, the public path is saved to the database along with the plain file name. These records are associated with the respective article or project to which the pictures are related.
Durring rendering, when the Markdown parser reads in an image tag, it will look for a relative path or a simple url. If it finds a relative path, then it looks up the related image by file name and renders the full, hosted, public path. This lets us author locally, and then render the picture from a performant image host.
One of the banes of my blogging existence until recently was the full width title images that I associated with each article. I had to find the image, crop it, save it, upload it to somewhere (often Evernote or Google Plus) then paste the url in the meta data of the post. What a bore.
Now the first picture with the file name of
title will be used as this header image for the article or the project. If there is no title picture, then the first picture associated with the document will be used, finally, there is a default header picture as fall back.
A few other header pictures for the static content pages are simply stored as Rails assets, with the whole lot of them synced to S3 as the asset host on deployment.
Reading about this project is not as much fun as watching it in action, it really is quite fast, even considering uploading some reasonable images. All of this is managed through free services attached to the Heroku instance, as well as AWS S3, and the Dropbox API. Enjoy: