Wednesday, 16 October 2019

Attaching a second VNIC card to compute in OCI

Well, just under 30 days ago, Oracle announced a series of resources you can use in OCI for free. One thing that had stopped me signing up and trying out OCI in the past was that I wanted to make the best use of the free credits, and knowing I wouldn't get a full chance to try things out in the 30 days, I didn't want to sign up prematurely. Now that they offer some free resources, this prompted me to sign up.

I am now at the end of the 30-day period where I have some credits to use non-free resources. One final thing I wanted to try out was attaching multiple VNIC (Virtual network interface card) to a single compute instance. One use-case of these is that you may want a machine accessible in 2 different networks.

It's not just a matter of attaching it in the OCI console - to bring the interface up you have to perform a couple of extra steps. When I was first trying this, I didn't read the docs and figured I would just have to edit the interface config script and bring it up, but no, this is not the correct method.

So first, create you instance. It's worth noting the free machine shape can only have 1 VNIC. Without upgrading your account, you will see you can allocate only 2 VNIC's, but if you looks at the documentation, it is certainly possible to have many more attached.

As a side note: At first I thought to assign a public IP address where you missed that step during the creation, I couldn't see the UI to assign a new one and thought I had to attach a new VNIC. Not the case - the setting is just buried deep!

On the instance page, there is an Edit VNIC link. However this is not where you can enable a public IP Address.



Instead, you have to go to the VNIC resource (go to the details page) and you will see a Resources section where you can update details about the IP address.



OK, back to the secondary VNIC. Back on the compute instance, under resources click Attached VNIC's and create a new VNIC. This will attach it to the server.

After you attach it, you will notice the new interface appear as one of your network devices, but without any IP address allocated.



Here, the interface we are interested in is "ens5".

Now, this is where we need to turn to the documentation. Here, they provide a script that you can run.

So, what we will want to do is login to the server as root, put a copy of that script and run it.


Perfect - all looking good. At this point if you reboot the server and check the IP information, you will notice it's not right - keeping the interface up hasn't persisted after a reboot.

There are a number of way you can configure this script to run at boot time, but for this example, I will leverage CRON. It has the frequency attribute of "@reboot" you can use to get a script to run whenever you boot the system.

So I would expect the crontab to have a line resembling:

@reboot /root/secondary_vnic_all_configure.sh -c

One thing you will also have to do is make sure /sbin is in your path as it calls a few commands in that directory and by default cron only includes /usr/bin and /bin.

And that's a wrap. You can reboot to verify, but otherwise your newly minted VNIC is all set up and configured.

Wednesday, 2 October 2019

OCI: Logging Object Events with the Streaming Service

There are two functionalities in OCI that we can leverage in order to support logging - Streaming and Events service.

With the events service we can define on which events to match whereby you specify a service name and the corresponding event types. So for object storage, we log events based on create, update and delete:


 The next part is that we can define the action type, with three possible options:

  1. Streaming
  2. Notifications
  3. Functions
For this article, we are looking into Streaming. So, the first step is to go ahead and make a stream. Nothing too complex here, just go to the Analytics, Streaming menu in the console, and create a new stream. When you create it, you specify a retention policy where it's defaulted to 24 hours. SO I will leave it at the default. Actually, I'm leaving it all at the default.

The next step is that we need to define an IAM policy so that cloud events can leverage this streaming functionality. So, head over to IAM and create a new policy with the text:

allow service cloudEvents to use streams in tenancy

You will want this policy in your root compartment.

Now, we can go ahead and create our event logging. Back over at Events Services (Application Integration -> Event Service), create a new rule. I called mine "StreamObjectEvents".

In the action, you want to specify action type as streaming and the specific stream events should go into. It should look like this:


 

With all that set up, go ahead and perform some operations on your bucket. Once done, head back over to your stream, and refresh the events, and you should see new rows in there.


Now that all the pieces are in place, it's time to figure out how we'll consume this data. In this example I'll be creating a bash script. It's a simple 3 part process:

Step 1 - We need to determine our stream OCID.

oci streaming admin stream list \
    --compartment-id $TS_COMPART_ID \
    --name ObjLog \
    | jq -r '.data[].id'


So here, I have my compartment ID set in an environment variable named "TS_COMPART_ID" and I want to get the stream with the name ObjLog.

Step 2 - Create a cursor

Streams have a concept of cursors. A cursor tells OCI what data to read from the stream, and a cursor survives for 5 minutes only. There are different kinds of cursors and the documentation kindly lists 5 types of cursors for us:

  • AFTER_OFFSET
  • AT_OFFSET
  • AT_TIME
  • LATEST
  • TRIM_HORIZON 
I found that AT_TIME returned logs after a given time, so I opted to use that type.

My code looks like this:

oci streaming stream cursor create-cursor \
    --stream-id $objLogStreamId \
    --type AT_TIME \
    --partition 0 \
    --time "$(date --date='-1 hour' --rfc-3339=seconds)" \
    | jq -r '.data.value'


Basically, I'm saying here I will want to get any events that occured since the last hour.

Step 3 - Reading and reporting the data

Now we have all the pieces, we can consume the data in our log. One note is that I think it would be better if this event data actually returned the user performing the action from an auditing point of view. Maybe it will be added in the future.

Also note that the data is encoded in base64, so we first need to decode it which returns JSON in a data structure that resembles the following:

{
    "eventType": "com.oraclecloud.objectstorage.updateobject",
    "cloudEventsVersion": "0.1",
    "eventTypeVersion": "2.0",
    "source": "ObjectStorage",
    "eventTime": "2019-10-02T01:35:32.985Z",
    "contentType": "application/json",
    "data": {
        "compartmentId": "ocid1.compartment.oc1..xxx",
        "compartmentName": "education",
        "resourceName": "README.md",
        "resourceId": "/n/xxx/b/bucket-20191002-1028/o/README.md",
        "availabilityDomain": "SYD-AD-1",
        "additionalDetails": {
            "bucketName": "bucket-20191002-1028",
            "archivalState": "Available",
            "namespace": "xxx",
            "bucketId": "ocid1.bucket.oc1.ap-sydney-1.xxx",
            "eTag": "bdef8e2e-fa20-4889-8cdc-fc1cb7ee5e3b"
        }
    },
    "eventID": "e8e5ef3b-1a98-4bf7-4e47-2827f517feae",
    "extensions": {
        "compartmentId": "ocid1.compartment.oc1..xxx"
    }
}

So, I iterate and output the data like so

tabData="eventType\teventTime\tresourceName\b" 
for evtVal in $(oci streaming stream message get \
    --stream-id $objLogStreamId \
    --cursor $cursorId \
    | jq -r 'select(.data != null) | .data[].value' \
    )
do
    evtJson=$(echo $evtVal | base64 -d)

    evtType=$(echo $evtJson | jq -r '.eventType')
    evtTime=$(echo $evtJson | jq -r '.eventTime')
    resourceName=$(echo $evtJson | jq -r '.data.resourceName')

    line=$(printf "%s\t%s\t%s" "$evtType" "$evtTime" "$resourceName")
    tabData+="$line\n"

done
 
printf "$tabData" | column -t 

I place this code on GitHub so you can see the complete code:

https://github.com/tschf/oci-scripts/blob/master/objlog.sh