Tuesday, 25 February 2020

Iterating OCI CLI list data in bash

For automating tasks with OCI, you have a few options:

  • OCI CLI (bash)
  • Python SDK
  • Go SDK
  • Java SDK
  • REST API

I'm a bash first kinda guy, so would usually opt for trying with the bash solution if the requirement is simple enough. Anything beyond, it's worth moving over to Python or GO.

When you run a command, you will typically get a JSON payload. But I'm in bash, how to I interact with this data?

That's where this nice tool jq comes in. An interface to JSON data, where you can pass in a json path to get the data you want. When you get started with this you will want to leverage the website jqplay.org. It provides a visual interface to the query paths you build, along with some common examples.

So, now over to iterating the data structure. I'm going to take the compartment list as an example.

When calling the command: oci iam compartment list, the data set in my tenancy looks like this:

{
  "data": [
    {
      "compartment-id": "ocid1.tenancy.oc1..xxx",
      "defined-tags": {},
      "description": "education",
      "freeform-tags": {},
      "id": "ocid1.compartment.oc1..xxx",
      "inactive-status": null,
      "is-accessible": null,
      "lifecycle-state": "ACTIVE",
      "name": "education",
      "time-created": "2019-09-20T01:06:31.731000+00:00"
    },
    {
      "compartment-id": "ocid1.tenancy.oc1..xxx",
      "defined-tags": {},
      "description": "idcs-xxx|22540605|foo@gmail.com-12345",
      "freeform-tags": {},
      "id": "ocid1.compartment.oc1..xxx",
      "inactive-status": null,
      "is-accessible": null,
      "lifecycle-state": "ACTIVE",
      "name": "ManagedCompartmentForPaaS",
      "time-created": "2019-09-17T02:56:55.916000+00:00"
    },
    {
      "compartment-id": "ocid1.tenancy.oc1..xxx",
      "defined-tags": {},
      "description": "Learning to use terraform",
      "freeform-tags": {},
      "id": "ocid1.compartment.oc1..xxx",
      "inactive-status": null,
      "is-accessible": null,
      "lifecycle-state": "DELETED",
      "name": "terraform",
      "time-created": "2019-09-25T12:29:43.421000+00:00"
    }
  ]
}

So, in my bash script, what I will normally do is get a list of indexes so I can look at these data sets one by one. To do these, you want to use the "keys" function which will turn an array of all the indexes. We want to remove the array brackets and just end up with a number on each line representing the index. So we end up with a json path of: .data | keys | .[]


And so when we are looping over our data, we just reference the index, to get individual properties for that element.

So, with all this info, our list script looks like this:

#!/bin/bash
set -e

compartmentList=$(oci iam compartment list)

for i in $(echo "$compartmentList" | jq '.data | keys | .[]')
do
    ID=$(echo $compartmentList | jq -r ".data[$i].\"id\"")
    name=$(echo $compartmentList | jq -r ".data[$i].\"name\"")
    desc=$(echo $compartmentList | jq -r ".data[$i].\"description\"")
    lifecycleState=$(echo $compartmentList | jq -r ".data[$i].\"lifecycle-state\"")

    echo "ID: $ID"
    echo "Name: $name"
    echo "Desc: $desc"
    echo "Desc: $lifecycleState"
    echo "****"
done

(side note: with this simple example, there's probbaly a one liner you could do with jq, but real world example are likely more complex and require some use of one or two properties)

Compartments usually underpin other operations you may be analysing in your tenancy, and one thing I discovered the other day is that if a compartment gets removed, things start going haywire! So, what we'll want to do is restrict our list to only include ones with the lifecycle-state of ACTIVE (depending of course on your business requirements).

So, in our script, we could just add a condition:

if [[ "$lifecycleState" == "ACTIVE" ]]
then
    # TODO
fi

However, just to revisit one of my previous blog posts which discussed the query capabilities of the command line client, we can reduce the code and create a reusable component/query.

Go to your oci_cli_rc file and add a new query called active, that looks like this:

active=data[?"lifecycle-state" == `ACTIVE`]

If we use this query in our command, it's worth noting that the data is transformed slightly. Previously it contains an array with the name "data". That property is removed when using this query and now we end up with a raw array, of objects.

So, if we want to use this query to filter out only active compartments, our script now looks like this:

#!/bin/bash
set -e

compartmentList=$(oci iam compartment list --query query://active)

for i in $(echo "$compartmentList" | jq 'keys | .[]')
do
    ID=$(echo $compartmentList | jq -r ".[$i].\"id\"")
    name=$(echo $compartmentList | jq -r ".[$i].\"name\"")
    desc=$(echo $compartmentList | jq -r ".[$i].\"description\"")
    lifecycleState=$(echo $compartmentList | jq -r ".[$i].\"lifecycle-state\"")

    echo "ID: $ID"
    echo "Name: $name"
    echo "Desc: $desc"
    echo "Desc: $lifecycleState"
    echo "****"
done

I made bold that parts that have been changed. You'll notice we just removed the reference to .data; Otherwise pretty well the same.

Friday, 14 February 2020

Oracle Cloud Infrastructure Cloud Shell - Here's what I discovered

This week I spotted a new terminal-like icon at the top of my OCI tenancy. Upload clicking it, its a GCP-esque terminal emulator directly in the browser. I was doing some tweets about my discoveries, but thought I'd be a good idea to collate those in once place in a more consumable place.

So, first I would say, it's possible you may not have it yet in your environment. At least for me, in my personal account, I do have it appearing. This was with a home region of Sydney.

Once you launch the shell, type help and you will see a link to the official documentation. You can find it here: https://docs.cloud.oracle.com/en-us//iaas/Content/API/Concepts/cloudshellintro.htm

Like in GCP, you get 5gb of storage. This will persist for at least 8 months. You get 6 months before OCI will email your tenant administrator and another 60 days after that before your data will be purged. So, a decent enough amount of time I would think.

You can verify this with the command: df -h.
On a fresh connection, I seem to have about 100MB used. Nothing to frown at.

In my fresh shell, the .bash_history file was not present. This meant your history wouldn't persist between sessions. Easy fix, just touch that file and then your history will persist between sessions.

In /home, there are two accounts:

1. Your own
2. oci

The OCI CLI client is installed to the oci user directory.
It is using a profile with whatever access you have in OCI that you are connecting from. The region is the one to which you connect from, although, you can easily switch to another region without connecting to a new shell after switching regions in the console. Just run the command:

export OCI_CLI_REGION=eu-zurich-1 # or whichever region you want

One change I'd suggest is something like the following into your `bashrc`.

export OCI_CLI_REGION=$OCI_CLI_PROFILE

(OCI_CLI_PROFILE is set to the region in which you are active in the console, and the bash prompt has the region hard coded into the prompt)

Then modifying your PS1 variable to reference OCI_CLI_REGION. The reason for this, is if you do modify the region variable, your bash prompt will be misleading and could lead to some confusion.

The documentation states the following are pre-installed:

  • Git
  • Java
  • Python (2 and 3)
  • SQL Plus
  • kubectl
  • helm
  • maven
  • gradle
  • terraform

Some useful information in relation to pre-installed software.

Git is version 1.8. That's not so old, released back in November, 2019. Just worth noting in case them are some recent feature set you're expecting. Current stable version is 2.25

Java version is 1.8

Python2 includes the oci SDK. Python3 does not.
If you try to install new packages, you will run into issues, so the best thing to do is create a virtual environment in your home directory and use that instead. Most especially if you wish to target python3, which you should be.
You can follow these steps:

cd $HOME
mkdir python3
python3 -m venv python3/
cd python3
bin/pip3 install oci



I would suggest then updating your path to point to this new $HOME/python3/bin folder so that then becomes the default python3 that your system uses.

SQL*Plus is the latest current release - 19.5.
SQLcl is omitted from the list. According to their social media account, this is something they're working to include - so keep your eyes out for that one!

Not mentioned, Golang is also installed. It is on version 1.13. Perfect!

Also not mentioned, Docker. It is also installed and is currently at 19.03

Minor software not mentioned, jq. This is a very useful tool for working with JSON on the command line. So goes hand-in-hand with the cli client.

There seemed be sufficient about of memory for any tasks:

              total        used        free      shared  buff/cache   available
Mem:           7.5G        650M        5.5G         16M        1.4G        6.6G
Swap:          8.0G          0B        8.0G


The documentation does mentions it times out after 20min of inactivity. This seems to be slightly flaky in my experience. Even though I've been executing commands, I've noticed I would still get disconnected.

Well, so far as looking pretty nice. Kudos Oracle!