Python, Flask, and Rest API

This blog post is a continuation of my Google Home\Google Action side project.  The first blog post is here.

The project keeps evolving the more I think about it.  The 2 diagrams below illustrate the workflows that need to be developed.

Populate the Database – Workflow 1

googlehome2

Google Home\Google Action – Workflow 2

googlehome3

So as you can see above, a critical piece of this puzzle is the REST API that needs to be created to allow us to interact with the data.

For the REST API, I have decided to use Python, Flask, and Flask-RESTful.  For this service I can host the API instance on Heroku if needed.  As for which database to use, I am thinking MongoDB, and if needed host it online with mLab.  Tools I will use to help get this API created are Postman, Curl, and PyCharm.  (Great resource)

First – Lets stub out the RestAPI code and the HTTP method “flask-restful” classes using PyCharm.

restmethods

Below is a “rough” first draft of the python REST API.

from flask import Flask
from flask_restful import reqparse, abort, Api, Resource

app = Flask(__name__)
api = Api(app)

# Sample lunch menu data for testing
LUNCHMENU = {
    '4202017': {'menu': 'Main entry is corn dogs'},
    '4212017': {'menu': 'Main entry is chicken nuggets'},
    '4222017': {'menu': 'Main entry is tacos'},
}

# Argument parser - must add all arguments explicitly here
parser = reqparse.RequestParser()
parser.add_argument('item')
parser.add_argument('date')

# Check if item exists and error if it doesn't
def error_if_item_doesnt_exist(id):
    if id not in LUNCHMENU:
        abort(404, message="Menu Item {} doesn't exist".format(id))

# Check if item exists and error if it does
def error_if_item_does_exist(id):
    if id in LUNCHMENU:
        abort(404, message="Menu Item {} exist".format(id))


# Define get, delete, and put method class
class MenuItem(Resource):
    def get(self, id):
        error_if_item_doesnt_exist(id)
        return LUNCHMENU[id]

    def delete(self, id):
        error_if_item_doesnt_exist(id)
        del LUNCHMENU[id]
        return '', 204

    def put(self, id):
        args = parser.parse_args()
        LUNCHMENU[id] = {'item': args['item']}
        return LUNCHMENU[id], 201


# Define list and post method class
class MenuList(Resource):
    def get(self):
        return LUNCHMENU

    def post(self):
        args = parser.parse_args()
        error_if_item_does_exist(args['date'])
        LUNCHMENU[args['date']] = {'item': args['item']}
        return LUNCHMENU[args['date']], 201


# Setup Api resource routing to classes here
api.add_resource(MenuList, '/menuitems')
api.add_resource(MenuItem, '/menuitems/<id>')

# If program is executed itself, then run
if __name__ == '__main__':
    app.run(debug=True)

Next step will be to true up the code to accept\parse the incoming JSON and then send the correct JSON format back to the API.AI agent.

  • Request format – https://docs.api.ai/docs/webhook#section-format-of-request-to-the-service
  • Response format – https://docs.api.ai/docs/webhook#section-format-of-response-from-the-service

New Docker\Ansible container per Playbook execution?

docker

What would it look like to setup a job in Jenkins that calls out to a Linux server, starts an Ansible Docker container that executes a playbook, and then shuts down the docker container like it never existed?  Time to find out…..

For this exercise, we are going to have a Jenkins slave installed on the same Linux server that we’re going to use to launch our Ansible Docker container.  Also, in our Jenkins job, we are going to configure Git as our source code repo, which is where out playbook and host files will be stored.  So when the Jenkins job is executed, the playbooks\host files will be copied to the slave.

We are going to follow these steps to setup the docker piece of this puzzle.

  1. Install docker on your Jenkins slave server and then verify your version.
    • (docker -v) = Docker version 17.03.1-ce, build c6d412e
  2. Download the following Ansible Docker image.  This image has an ENTRYPOINT of “ansible-playbook” and a WORKDIR of “/ansible/playbooks/.  (This is key!!!)
  3. Now we need to make note of the location Jenkins downloads the Git files to on the slave.
    • Example: /home/build/ansible_git/myPlayBook.yml
    • We will use this location when starting up the docker container.  (-v)
  4. Running the following command on the Jenkins slave, should then start up the Docker container that has Ansible installed, link the local folder “/home/build/ansible_git” to the container folder “/ansible/playbooks”, execute the playbook, and then shut everything down.
    • docker run –rm -it -v /home/build/ansible_git:/ansible/playbooks <image_id> myPlayBook.yml
      • –rm = cleanup
      • -it = interative and ttl
      • -v = volume (local host:container)

Python, Google API, and PDF

As a side project, I was thinking about creating a Google Action that returned the daily lunch menu for my children’s school from Google Home.  This would surely come in handy for my family and others.

To complete this project, I was thinking the following tasks would need to be completed.

  1. Find a solution to getting the daily lunch menu via Python (This blog post)
  2. Store the lunch menu data is some location\format so my Google Action can read it
  3. Create Google Action (Create only in my local test environment…)

This blog post goes over what I did for step #1.  There may be easier ways to do this, but it was a good learning experience.

So, the school lunch menu is stored in a PDF file on Google Drive.  I first need to find a way to automatically download this PDF file, read the contents of the PDF, and then parse out the necessary lunch menu information for use.

Step #1 – Download the Lunch Menu PDF file from Google Drive

  1. We will use the Google Drive API and Python to download the PDF file locally.
    • Note that to store the data locally we will need to use “io.FileIO” and not “io.BytesIO” from the provided example in the above link.
  2. You will need to activate the Google API with your projects correct OAuth credentials.  Follow the steps in this link as needed.  This link will also show you how to install the Google API python package.
  3. Get the Google Drive ID associated with the file you wish to download.
    • You can get your file ID from a shareable link to the document.  To get a files shareable link, go to Google Drive, select your file, and then click “Get shareable link” which will return a URL that will contain the ID.
    • googleapilink.png
  4. Create your Python script to download the file.
from __future__ import print_function
import httplib2
import os
import io

from apiclient import discovery
from oauth2client import client
from oauth2client import tools
from oauth2client.file import Storage
from apiclient.http import MediaIoBaseDownload

CREDENTIAL_DIR = './credentials'
CREDENTIAL_FILENAME = 'drive-python-quickstart.json'
CLIENT_SECRET_FILE = 'client_secret.json'
APPLICATION_NAME = 'Lunch Menu Download'
SCOPES = 'https://www.googleapis.com/auth/drive.readonly'
FILE_ID = 'Your File ID Here'
DOWNLOADED_FILE_NAME = 'lunch.pdf'

def get_credentials():
  credential_dir = CREDENTIAL_DIR
  if not os.path.exists(credential_dir):
    os.makedirs(credential_dir)
  credential_path = os.path.join(credential_dir, CREDENTIAL_FILENAME)
  store = Storage(credential_path)
  credentials = store.get()

  if not credentials or credentials.invalid:
    flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPES)
    flow.user_agent = APPLICATION_NAME
      if flags:
        credentials = tools.run_flow(flow, store, flags)
      else:
        credentials = tools.run(flow, store)
  return credentials

def main():
  credentials = get_credentials()
  http = credentials.authorize(httplib2.Http())
  service = discovery.build('drive', 'v3', http=http)
  file_id = FILE_ID
  request = service.files().get_media(fileId=file_id)
  fh = io.FileIO(DOWNLOADED_FILE_NAME, 'wb')
  downloader = MediaIoBaseDownload(fh, request)
  done = False

  while done is False:
    status, done = downloader.next_chunk()
    print('Download %d%%.' % int(status.progress() * 100))

if __name__ == '__main__':
main()

 Step #2 – Parse the PDF file that you just downloaded

  1. To parse the PDF file with Python, we will use the package Slate.  The Slate package will basically return a Slate object that contains all the text from the PDF file.
  2. The text returned was “for the most part” in a structured order with newline characters separating the contents.
  3. I decided to split the text returned by “\n\n” and use the order to pull out the necessary information.
    • pdf_text_split = stringpdf_text.split(“\\n\\n”)
  4. Now I step through each item in the list that was returned and place it into 1 of 4 lists based on some compare logic.
    • Main Entry #1
    • Main Entry #2
    • Main Entry #3
    • Snack
  5. Once I parsed all the data from the split and placed it in the correct list, I combined all the individual lists into a 4-dimensional list called Lunch Menu
    • lunchmenu=[mainentry1,mainentry2,mainentry3,snack1]
  6. Now I can retrieve the items from the lists with ease.
    • lunchmenu[1][12] would return the 13th “Main Entry #2” item.  The #13 would represent the 13th day of school for a particular month, not the 13th day of the month. (M = 1,  F = 5)
    • In my case, the result is “Beef Taco Salad” for the second main entry on the 13th day.
  7. Below is the Python code that performs the above actions.
# Import slate python package for extracting text from a PDF file
import slate

# Open PDF file
with open('lunchmenu.pdf', 'rb') as f:

# Generate text from PDF file
pdf_text = slate.PDF(f)

mainentry1 = []
mainentry2 = []
mainentry3 = []
snack1 = []
lunchmenu=[[],[],[],[]]

stringpdf_text = str(pdf_text)
pdf_text_split = stringpdf_text.split("\\n\\n")
index = 0
for food in pdf_text_split:
  if"1)" in food:
    mainentry1.append(food)
  if"2)" in food:
    mainentry2.append(food)
  if"3)" in food:
    mainentry3.append(food)
  if "Snack" in food:
    snack1.append(food)
  index = index + 1

lunchmenu=[mainentry1,mainentry2,mainentry3,snack1]

# Persist the lunchmenu list here or do something else...
# Should close the file handle also...(f.close())
So the next step might be to persist the data somewhere instead of creating lists and then determine the proper way to serve it up to a new Google Action.

VSTS For Family Planning? Get out!

Visual Studio Team Services (aka VSTS) is the online\cloud hosted version of Team Foundation Server.  There are some differences\benefits between the 2 which Buck Hodges clearly explains here.

Microsoft offers free access to VSTS for smaller teams. (5 users)  This free access will allow you to get a pretty good handle on the VSTS online tool and its features.  I must say I am thoroughly impressed.  I have been a TFS user since the 2005 days and appreciate how far the tool has come.

So, after playing around with the tool I started pondering the idea of using it to help plan family tasks every week/month.  By doing this, my family could get exposure to task boards, burn-down charts, stories, etc…  So without even knowing it, they would be learning the ways of Scrum.

Below is a quick mock up of a family task board for an April iteration.

VSTS.png

VSTS offers the ability to setup notifications and also allows you report off of past tasks.  I can see both of these coming in handy.

You can also get VSTS on your mobile device, although I would consider this app useless for this family project.  Seems to be geared for a PM or manager.

If I decide to give VSTS a try for family planning, I will be sure to provide the results here.  🙂

Azure Storage Emulator & Explorer

Want to easily test\develop against Azure Storage Services locally?  If so then you should try the new Azure Storage Explorer app and the Azure Storage Emulator!  For reference, the following storage services are currently offered by Azure.

  • Blob (Container) storage service
  • Table (NoSQL) storage service
  • Queue (Messaging) storage service
  • File (Share) storage service

You can test 3/4 of the storage services locally (free) before implementing any changes in Azure.  Using the emulator, you can simulate Blob, Queue, and Table services locally.

Azure Storage Emulator

Setting up the emulator is pretty straight forward.  Follow the directions here.  After the install, you can write your storage code and target the emulator.  Quick “Create Blob Container” python script below. (pip install azure==2.0.0rc)

import random, string
from random import randint
from azure.storage import CloudStorageAccount
from azure.storage.blob import BlockBlobService

account = CloudStorageAccount(is_emulated=True)
blockblob_service = account.create_block_blob_service()
container_name = 'samplecontainername'
blockblob_service.create_container(container_name)

Azure Storage Explorer

Now you can install the storage explorer application to view the storage data.  From the application, you will be able to view Azure storage in the cloud (SAS) and your emulator storage services.

AzureStorage

I think the combination of the emulator and the storage explorer make for a nice cost-efficient development environment…

Ant vs Maven vs Gradle

If you are working with Java projects, then you are probably using one of the following 3 build tools.

The majority of my experience has been with Ant, which isn’t surprising since its been around the longest.  I’ve also used both Gradle and Maven for building Java projects.

Below are some of my thoughts on the benefits\draw-backs of each.  For a really detailed comparison of each tool, check out this article!

ANT 

This is my favorite build tool!  It’s probably because I am so familiar with reading Ant XML files. 🙂  I personally like working with XML files.  With Ant, there is no preconceived folder\file structure that your project needs to be in.  You have the ability to set source location properties in your Ant XML file as needed.  There are also many standard “Tasks” that you can use, along with bonus “Contrib” tasks and even “AWS” tasks.  For dependency management, you will need to use Apache Ivy in you projects build scripts.

My biggest concern with Ant is the over-all usage of the build tool is going down, which means less contributions from the community.

Ant Example – Compile Java Source (No dependency because its to much to add!!)

<project default="compile">
 <target name="compile">
  <javac srcdir="src" destdir="target"/>
 </target>
</project>

Maven

I’m a fan of Maven too!  This tool is by far the most popular of the 3.  Unlike Ant, Maven can handle both the build and dependency management itself.  One of the biggest differences from Ant is the “convention” aspect.  With Maven, your project is usually setup following the standard layout.  In a lot of ways this is good for new projects.  If you are trying to migrate an older project to Maven, this could be bad.  My suggestion with Maven is really try to understand the standard project layout before trying to use it to build your projects.

Like Ant, Maven uses XML to define the build process.  Maven XML build files are called POM.xml files.  These files assume standards and can get extremely large.

Maven Example – Compile with JUnit dependency

<project>
 <modelVersion>4.0.0</modelVersion>
 <groupId>com.devopsunleashed.blog</groupId>
 <artifactId>search</artifactId>
 <packaging>jar</packaging>
 <version>17.3.1</version>
 <name>search</name>
 <url>http://maven.apache.org</url>
 <dependencies>
 <dependency>
 <groupId>junit</groupId>
 <artifactId>junit</artifactId>
 <version>3.8.1</version>
 <scope>test</scope>
 </dependency>
 </dependencies>
</project>

Gradle

This tool is the newest of them all and is gaining traction!  The first thing I like about Gradle is the ability to do incremental builds.  In some cases, saving 5 minutes of compile time can really add up in terms of $$$.  Another bonus is the ability to ingest Ant scripts into the Gradle build process.  Finally, like Ant, Gradle does not have to follow a standard project convention, although it does make it a lot easier!

Gradle Example – Compile with JUnit dependency using standard convention

apply plugin: 'java'

dependencies {
 testCompile 'junit:junit:4.12'
}

repositories {
 mavenCentral()
}

I will say that pulling in dependencies is pretty straight forward with Gradle!

maven

Docker CE vs Docker EE

Big announcement from Docker!  Looks like we have some new product names in the house.  (Docker CE & Docker EE)

  • EE = Docker Enterprise Edition (Previously Docker Commercially Supported (CS))
  • CE = Docker Community Edition (Previously Docker Engine)

After you read the article above, you will notice some benefits to this new approach.

Below are some of the benefits I see.

  • A clear release cadence for both EE and CE.
    • EE will follow the stable CE releases

EECE2.png

  • New time-based versioning scheme (17.03.1, 17.03.2, 18.02.1).
    • So easy to understand now!
  • Ability to separate Community (CE) updates from Docker (EE) updates .
    • Good!! – If all good features get put into CE, and not just EE!!
  • Allowing 3rd party vendors to certify their containers and plugins for Docker EE.
    • Good for Enterprises that need re-assurance!!

EECE.png

Something to point out from the above article is Docker CE and RHEL do not seem to be supported.  However, using these CentOS install instructions, I was able to get it running as a quick test.  (Fedora -> RHEL -> CentOS)  

docker2.png

However, per the documentation, RHEL and CE do not align with the support matrix below.  Also, it sounds like you could contact Docker about licensing EE for development purposes which seems to be what this article implies.

Docker.png

Sensu Stack – Beginner Notes

What is the Sensu Stack?  For a detailed explanation, read this webpage.  The Sensu stack has multiple components, which generally consist of the following.

  • Sensu Client – This is the agent that collects data.  Like Telegraf, the Sensu Client can be installed on both Linux and Windows.  
  • Sensu Server – This is where the data from the client is processed and then pushed to the data layer.
  • Sensu API – Allows access to data from tools
  • RabbitMQ – This is the transport layer, responsibly for the communication between Client\Server.
  • Redis – Data persistence layer.
  • Uchiwa – Dashboard for viewing data

Below are some helpful notes for getting the Sensu stack components setup and running.

Helpful Links

Sensu Configuration Loading Sequence

  • Review the Sensu configuration property loading cadence here.

Redis Configuration File

  • /etc/redis.conf (Update this file to change host\port\etc….)
    • Default IP and Port = 127.0.0.1:6379

RabbitMQ-Server Configuration File and Management UI

  • /etc/rabbitmq/rabbitmq.config (Update this file to change host\port\etc….)
    • Default IP and Port = 127.0.0.1:5672
  • Managment console
  • rabbitmq

Sensu Server Configuration Files

  • /etc/sensu/config.json (Main configuration file – Files below are specific and get merged with main configuration file)
  • /etc/sensu/conf.d/api.json (Sensu server address, API bind address\port)
  • /etc/sensu/conf.d/client.json  (Contains client name, address, and subscriptions)
  • /etc/sensu/conf.d/rabbitmq.json (Contains the host\port for the rabbitmq server instance)
  • /etc/sensu/conf.d/transport.json (Contains the transport solution)
  • /etc/sensu/conf.d/redis.json (Contains the host\port for the redis instance)

Sensu Client Configuration Files

  • /etc/sensu/conf.d/client.json  (Contains client name, address, and subscriptions)
  • /etc/sensu/conf.d/rabbitmq.json (Contains the host\port for the rabbitmq server instance)

Uchiwa Configuration Files

  • /etc/sensu/uchiwa.json

General Log File Locations

  • Sensu Server = /var/log/sensu
  • Sensu Client = /var/log/sensu
  • RabbitMQ = /var/log/rabbitmq
  • Redis = /var/log/redis
  • Uchiwa = /var/log

Service Start\Stop Commands

  • service sensu-client start
  • service sensu-server start
  • service sensu-api start
  • service uchiwa start

Testing the Stack

  • If you are unable to access Uchiwa on port 3000, you may need to disable your firewall or at a minimum open port 3000
    • sudo firewall-cmd –zone=public –add-port=3000/tcp –permanent
  • You should be able to target the Uchiwa instance via http”\\<server_name>:3000
    • Example: http://<server_name>:3000/#/clients
  • If everything is linked together properly you will get a nice gui like this!

 

uchiwa.png