Working at the Speed of News

Day 3 ~ April 9, 2019

Overview

"Laziness: The quality that makes you go to great effort to reduce overall energy expenditure..." ~ Larry Wall, creator of Perl

When news breaks and you're new to a beat, it can be especially stressful to hunt down sources and pull together copy under deadline. Veteran reporters, on the other hand, often have (slightly) lower stress levels. They typically have a wealth of domain
knowledge to dip into. They're intimately familiar with the trends and key players on their beat, as well as documents that can provide background or help answer questions. They have a stable of human sources who can speak to an issue or point them toward other, more relevant sources.

In a similar vein, news app developers just starting out often work in ad hoc, "messy" ways. This is natural when you join a team, learn a new technology, etc. But over time, news app developers also build up a stable of tools and resources to help them work at the speed of news. Developers -- in the news and beyond -- make a virtue of "laziness."

Identify workflows to automate

This is not to say we're prepared for every situation. News breaks in unexpected ways and the tools and strategies called for on one day will differ from those required on another. A critical role for the news app developer is to identify patterns in the constant thrash of breaking news, and strive to build tools to help streamline similar situations in the future. If reporters in your newsroom deal with large document dumps on a regular basis, perhaps you need a custom app to sift through and make sense of documents. If your organization frequently hand-keys data from government documents, it may be worth leveling up on a web framework that lets you easily build custom data admins.

To work at the speed of news, developers automate common workflows and build tools to help simplify the work of their newsroom colleagues. Here are a few examples of automation from the news industry:

Build incrementally, with partners

So you've identified a way to help colleagues through automation, and the newsroom is itching to get their hands on the tool.

Before rushing a project out the door, remember that it's difficult to get it "right" on the first pass -- especially if you go into a coding silo for weeks or months. When building a new tool, we're often learning new technologies -- APIs, code libraries, even new languages. Initial work can be slow and the first version of our code inflexible, messy or non-idiomatic. Often, we discover nuances about a workflow only after living with a tool for some time. It's quite easy to create a hard-to-change prototype that fails to address the pain points that inspired the project.

We can address many of these issues by choosing to build incrementally, in partnership with end users who best understand the needs the tool seeks to address. Bring reporters and editors into the development process. Form a design committee and ask users to help define and prioritize features. Hold weekly demos to show off the latest work and get feedback along the way. By engaging users in the development process, they'll gain an appreciation for the time it takes to build useful, reliable software. They'll help identify features and nuances you never considered. They'll also be more invested in the finished product since they helped craft it.

It's worth noting that these are not new ideas. A buzzword-y industry and elaborate coding practices have evolved around this type of approach, but the underlying principles remain valuable.

Hands-on Exercise

In the spirit of simplifying workflows, we'll continue our website-building exercise by automating the manual labor of uploading new pages for our website. This will require leveling up on the AWS Command-Line Interface and then "wrapping" the AWS invocations -- which can grow unwieldy -- in human-friendly commands using a Python task-execution framework called invoke. This way, our brains can spend less time on the boring stuff and focus more on building our site and sundry applications.

Setup

All this work should be done in the private Github repo we created for class in Week 1. Please ensure I've been added as a
collaborator on the repo so I can access your work.

Some preliminary installs and setup:

Install and configure the AWS CLI. If you run into trouble, see these instructions.
Install Pipenv: pip3 install pipenv

Install invoke (a Python command-line framework):

cd /path/to/<comm177t-code-repo>
pipenv install invoke

Create a directory to hold your site pages:
```
mkdir _site/
```
Add the basic "hello world" index.html page to the _site/ directory (e.g. the index.html we created and uploaded in Create a Static Website).
Commit all the new files (Pipfile, Pipefile.lock, _site/index.html) to your repo and push them to Github

Automation exercise

The AWS Command-Line Interface provides a myriad of tools that let you programmatically control the many services they offer.
For example, we can use the AWS Certificate Manager interface to manage SSL certificates (this assumes you've completed Adding a Custom Domain and HTTPS to Your Website):

aws acm --profile stanford list-certificates

For this exercise, we'll learn how to deploy our website pages and ensure that stale versions of files are cleared from the web. As with any technology, part of the learning curve involves learning how to read technical documentation.

For this exercise, you must reference the AWS Command-Line Interface docs and do the following:

Identify the appropriate AWS CLI command to push local files from the _site/ directory to S3. The directory name (_site) should not appear in the bucket -- only the files/folders contained within it.
Ensure that files which have been deleted locally are also deleted on S3.
Find a way to invalidate, or clear, stale versions of files from CloudFront

Assignments

Assignment 3

On Thursday, April 11, Youyou Zhou of Quartz will visit to give us the back story on the creation of Hurricane Maria's Dead. This award-winning project was a year in the making and involved a large-scale collaboration between multiple news organizations, college students and volunteers. Spend some time reviewing this interactive and submit two questions via Canvas before our speaker arrives.

Assignment 4

To complete this assignment, you must commit your invoke script and default index.html page to your Github repo and upload a link to the repo via Canvas. Please make sure I've been added as a collaborator on your repo, otherwise I won't be able to view the code. If I'm unable to view your code by the deadline, it will result in a penalty for lateness.

Before starting the code, it's worth reading through the Getting Started docs to get your bearings.

For this assignment, create an invoke script at the root of your repo called tasks.py containing two commands: deploy and cachebust. These commands should "shell out" to the AWS command-line invocations we crafted during the hands-on exercise in class. For details on how to execute local shell commands using invoke, see the Getting Started docs.

If you configured the AWS CLI with an administrator profile (in addition to a root account set up as the default), you must specify the profile in the AWS command-line invocation (see the above aws acm example).

Here are the requirements for the commands:

deploy - An AWS command that pushes static files to your S3 bucket. The directory name (_site) should not appear in the bucket -- only the files/folders contained within it. Also, the script must delete files in the S3 bucket if they've been deleted locally. You should carefully review the command's options to figure out which is needed to implement this behavior.
cachebust - An AWS command that will invalidate, or clear, all assets on AWS CloudFront.

In order to run these commands, you must "activate" the virtual environment for the repo. This environment was created automatically when you ran pipenv install invoke.


cd /path/to/<comm177t-code-repo>

# Activate the environment
pipenv shell

# Use invoke
invoke --list
invoke deploy
invoke cachebust

If you're new to pipenv, check out the example workflow docs.