Crontabs: Automating Python Scripts

I’m very excited to have created my first python command line package! I call it the ‘Sensor to CloudantDB Cronifier’. Cool name, right? It can do the following:

• create a DB
• load data into DB
• query data
• list all DB
• delete a DB

The command line tool

To run a cron job, you give it only 3 variables: the DB name, the duration in minutes (how long you want to run the job for), and the interval (how many seconds in between each ping.) In this case, I’m not getting the data from the sensor via MQTT. I’m polling an API endpoint for the data. For example:

python package.py awesomeDatabase 60 5

This command tells the script to ping the endpoint every 5 seconds for one hour and dump the data into awesomeDatabase.

To turn it into a command line tool, I used an old school technique: sys.argv. I know, I should use something cool and sexy like Click. But to my defense, it works just fine. Just look how simple:

import sys
if sys.argv == 'runcron':
     runcron()

The Problem

Even though the command line package is awesome, one has to type in the command and your terminal window is totally booked (you can’t use it) until the job is finished. And, once the job is finished, you’ll have to manually run the command again. So — it’s not that smart! I asked myself, how do I automate this task so that it will run in the background continuously.

On macOS and linux, there is a standard package called crontabs. Crontabs automate your scripts on your machine. A quick google will show you that it’s a popular tool for people to back up their personal database with a shell script or use it with a raspberry pi. In this case, I’m going to run it simply on my macOS.

Setting up Crontabs

In your terminal, run env EDITOR=nano crontabs -e

This will open up the nano editor so you can create a cron job.

Running crontab -e opens the vim editor.

A cron job looks like this:

* * * * * cd /path/to/script/directory && /path/to/python3 /path/to/script 

Wow, that’s a long command. Let’s break it down! The 5 asterisks is where Cronos lives — the greek god of time. The stars from left to right represent: minute(0–59), hour(0–23), day of month(1–31), month(1–12), day of week(0–6). Instead of fiddling around (I would’ve loved to but my company bills by time), I used this handy crontab generator to get the right time code.

A quick rule of thumb: ***** is once every minute. */5**** is once every 5 minutes.

The next line cd /path/to/script/directory points to where your script directory lives. It’s important that it’s the directory and not the script itself. Because at this point, crontab does not know to use python. That’s where the next line comes in. /path/to/python3 is the equivalent of running python3.

If you don’t know where your python3 is living, I used the command find / -name python3. It greps your entire machine for anything matching the name ‘python3’. (you can run the command with sudo to give it permissions to search through more stuff but I doubt you will need that level of privilege.) I found mine under:

/Library/Frameworks/Python.framework/Versions/3.6/bin/python3

Finally, the last set of command points to the actual script.

At this point, you can add additional stuff like, maybe you want to pipe the log files. At the end of this command, add this >> /path/to/file.log And the logs will be saved here.

To view your cronjobs, run crontab -l

You should be able to see your script. If the script worked, you will get an email. You’re like, “What? I get an email??” I know, it’s wild right? A cronjob by default sends a log via email to your machine. If you don’t have an email setup, it will log ‘You’ve got mail’. You can view the mail at the directory /var/mail/<your username> . It will return some crontab data. If there was an error or if your python script prints anything out, it will be spit out here. If you want to turn the mailing feature off, go back to your cron script and enter MAILTO=""

Voila

Small things: I wanted crontab to use my python script’s command line interface. I played with passing a variable in via crontab. Because, ideally in the future, I don’t want to have to go into the python script to make changes — I want to jump into the crontab editor and switch it, to another database for example. But (again it’s the billable hours), I took the shortcut and set the command under __main__ like so:

if __name__ == "__main__":    
    run_cron("awesomeDatabase")

In the future, I want to programmatically designate which database.

And that’s the end of my crontab adventure! =)