EL:DIABLO

Event data in a box, basically.

About

The main goal of EL:DIABLO is to overcome dependency hell and make it much easier for the end user to get up and running with event data coding. Using software designed for development operations allows us to easily share the setup we use to develop the tools and software for creating and working with event data. In short, no matter what hardware or operating system a user chooses, it is possible to replicate our exact event-data coding platform on that specific configuration. This goal is important to us for two primary reasons. First, we are striving to make the generation of event data more open than it has historically been. Things such as copyright and licensing agreements make it difficult to share source texts for coded event data, but we can make the process as transparent as possible. This is especially important since there are a multitude of seemingly minor choices that go into event data coding that can have a significant impact on the final product. The second reason we are pursuing EL:DIABLO as a project is to enable collaboration. It will no longer be unclear what steps are taken to generate event data, or what the various moving pieces within the system are. If someone wishes, for example, to develop a new event coder, they can simply drop that in to the existing pipeline. The same holds for the various dictionaries, geocoders, web scrapers, etc. It's like Legos. But for event data.

Components

On the technical side of things, EL:DIABLO provides the information and scripts necessary to set up a virtual machine on a user's computer. For those not familiar, this can be thought of as a computer within a computer. EL:DIABLO relies on Vagrant, and by extension VirtualBox, to set up this virtual environment. These two pieces of software allow for the easy setup and use of a virtual machine. Thus, two of the files contained within EL:DIABLO are a Vagrantfile, which gives instructions to Vagrant on how to setup the virtual machine, and bootstrap.sh, which is a shell script that installs the necessary software within the virtual machine.

The EL:DIABLO event coding platform is comprised of two primary applications: a web scraper and a processing pipeline (scraper and phoenix_pipeline specifically). The scraper is a simple web scraper that makes use of a whitelist of RSS feeds to pull stories from popular news outlets. The pipeline moves the news stories from storage in a database to the event coder, such as TABARI or PETRARCH, and outputs event data. More information about the details of these projects can be found in their respective documentation, linked to above. If you use the standard bootstrap.sh script provided with EL:DIABLO, the web scraper will run once an hour, and the pipeline will run once a day at 01:00.

Setting up

As mentioned above, EL:DIABLO relies on Vagrant and VirtualBox for most of the heavy lifting. This means that the only things that a user needs to install on their local machine are these two pieces of software. The creators of this software describe the install process better than we can, so a user should look here for Vagrant and here for VirtualBox. Once that software is installed, EL:DIABLO needs to be downloaded from the Github repository. For those familiar with git, a git clone should work fine. For those unfamiliar with git, it is possible to download the repository as a zip file as shown in the picture below.

Once this file is downloaded and unzipped, you should use the command line to cd into the directory and do vagrant up. This will take awhile to download the operating system image (this will only be done once) and then install the relevant software within the virtual machine. Seriously, this is going to take time; the process hasn't stalled out. Then vagrant ssh to get into the box. You're now in the virtual machine. Overall, this should look something like:

As a note, all of this will create a folder somewhere on your local machine that contains the operating system images. On OSX it's in the home directory and named VirtualBox VMs.

To get out of the virtual machine, type exit, which will bring you back to your local machine. There are three methods for ending the Vagrant box: vagrant suspend, vagrant halt, and vagrant destroy. The main difference between these three is the amount of system resources used while in the "down" state. If you are completely done with the virtual machine, and do not wish to keep any of the data, make use of vagrant destroy. Again, this will remove all of the data within the virtual machine and all software will have to be reinstalled. If you wish to just temporarily bring down the virtual machine, the other two commands should be explored in the Vagrant documentation.

Other Information

Due to the way Vagrant sets up the virtual machine, it is necessary to prepend nearly every command with sudo.

The bootstrap.sh script is specifically configured for use with the Vagrant box, but with slight modifications can be used on any Linux box (it's what we use to bootstrap our machines). This means that the script can serve as the basis for setting up a high-performance computer running EL:DIABLO, an individual's laptop, etc.

For the two Github repositories, scraper and phoenix_pipeline, each time vagrant up is run the most recent version of the code is pulled from Github. If you have a long-running virtual machine and wish to obtain the latest code, you can cd into the appropriate directory and run sudo git pull.