Monday, 22 May 2017

SQLcl scripting and npm modules

Node, or rather npm, has a huge collection of submitted modules! And if you've been using SQLcl to connect with the database, you should know it has scriptable ability where you can for example run JavaScript code and Java APIs. So the question is, how do we get the script to pick up node modules?

During my research, I found a few suggested methods:
  1. J2V8
  2. Project Avatar
  3. jvm-npm
I didn't try the first two methods, but have had success using the third (jvm-npm). This is a library with the source code hosted over on GitHub - https://github.com/nodyn/jvm-npm.

At this point, it's worth noting - not everything will work, as it does not include the full NodeJS API. 

So, first step to do is you'll probably want to host that library somewhere in your project - in my example, which I will provide the link to at the end of the article, I place it at ./lib/jvm-npm.js. Worth mentioning, you can also source this over http in your script, but probably best to keep a local copy.

For this example project, I'm going to focus on the lodash node module. So, the next step will be to set up a package.json file where you list any node dependencies - this is purely so I can run `npm install` to install any dependencies my script requires. So go ahead and run `npm init` in your project directory. After completing the wizard add your dependencies. You should end up with something like:

{
  "name": "sqlcl-npm-demo",
  "version": "1.0.0",
  "description": "Example of using node modules in an SQLcl script",
  "main": "lodashExample.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Trent Schafer",
  "license": "Apache",
  "dependencies": {
    "lodash": "^4.17.4"
  }
}


Now, after that is done, run: `npm install`.

trent@birroth:~/Projects/sqlcl-npm-demo$ npm install
sqlcl-npm-demo@1.0.0 /home/trent/Projects/sqlcl-npm-demo
└── lodash@4.17.4
trent@birroth:~/Projects/sqlcl-npm-demo$ ls node_modules/
lodash


Ok, so we have all of our dependencies installed. Now, how to we get this to work in our SQLcl script?

In a typical node app, you would have something like:

var _ = require('lodash');

If you try this out now, you will get an error like so:

SQL> script lodashExample
javax.script.ScriptException: ReferenceError: "require" is not defined in  at line number 1
<stack trace trimmed>


So, in order to get `require` available to us, we need to make it so our script has this jvm-npm library loaded - that we place in our project earlier. If you're not aware, you can load other scripts into a nashorn script with the `load` function. Read more over at this article: https://wiki.openjdk.java.net/display/Nashorn/Nashorn+extensions.

So, at the start our script, add the following call:

load('./lib/jvm-npm.js');

Then try re-running your script, and you should notice no errors this time around. So, it looks like we are good to go! Just to verify, let's take something out of the lodash API to see if all is working as expected. Lodash has an `nth` function that basically retrieves the specified index of the passed in array - with one little enhancement, it supports negative indexes to search backward - so let's try that out.

My full script becomes:

load('./lib/jvm-npm.js');
var _ = require('lodash');

var testArray = [
    55,
    12,
    99,
    65,
    164,
    32
];

var secondLast = _.nth(testArray, -2);

print("The second last item was", secondLast);


Then, the test output:

trent@birroth:~/Projects/sqlcl-npm-demo$ sql vmtest/vmtest@//192.168.1.116/xe

SQLcl: Release 4.2.0 Production on Sun May 21 19:40:07 2017

Copyright (c) 1982, 2017, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production


SQL> script lodashExample.js
The second last item was 164


Which all looks correct. Yay, success.

As I mentioned, it doesn't include the full node API, so you will probably need to be cautious going forward if you use this - but just offers that little bit more.

Working files from this example: https://github.com/tschf/sqlcl-npm-demo

Monday, 15 May 2017

Loading data from a web service. An alternate approach

I was recently working on a project where my client had just implemented a new system, and now they wanted to utilise the provided web services on this platform for some more reporting that wasn't provided out of the box. Being we use APEX, we had the nice APEX_WEB_SERVICE API to use - which made things nice and simple, but before that we needed to perform a couple of extra steps, which is required each time you want to pull data over the web from another domain.

It's actually, I feel, too complicated to get going - but I understand these restrictions are in place to keep systems nice and secure.

In a nut shell, we should be dealing with resources secured with some certificates - so we need to create both an ACL to the resource followed by creating a wallet with all the certificates the site uses. After that, we are good to go - until some time down the track when the service updates there certificates, then we once again need to update our wallet with the sites certificates.

Now, we got everything all working - the web service returns an XML feed. We parse the result and load it into tabular format into one of our tables. This was set up to run once a day - real time data isn't a strong requirement.

Given we are only running this once a day, I thought another approach could be the leverage SQLcl. It supports running Java and JavaScript through it's scripting engine. So, first I fired up my Java IDE to prototype performing a GET request, and came up with the following:

CredentialsProvider provider = new BasicCredentialsProvider();
UsernamePasswordCredentials creds = new UsernamePasswordCredentials("fakeuser", "fakepassword");
provider.setCredentials(AuthScope.ANY, creds);

HttpClient client = HttpClientBuilder.create()
        .setDefaultCredentialsProvider(provider)
        .build();
HttpGet get = new HttpGet("https://example.com/path/to/resource");

try {

    HttpResponse resp = client.execute(get);

    System.out.println(resp.getStatusLine().getStatusCode());

    HttpEntity entity = resp.getEntity();

    String respBody = EntityUtils.toString(entity);
    System.out.println(respBody);

} catch (IOException e ) {
    e.printStackTrace();
}


That worked well, so now I needed to convert this into a script that could be used with nashorn. So I figured out all the classes I needed, and added the following to a JavaScript (effectively, import statements):

//All the required Java clasees
var CredentialsProvider = Java.type("org.apache.http.client.CredentialsProvider"),
    BasicCredentialsProvider = Java.type("org.apache.http.impl.client.BasicCredentialsProvider"),
    UsernamePasswordCredentials = Java.type("org.apache.http.auth.UsernamePasswordCredentials"),
    AuthScope = Java.type("org.apache.http.auth.AuthScope"),
    HttpClient = Java.type("org.apache.http.client.HttpClient"),
    HttpClientBuilder = Java.type("org.apache.http.impl.client.HttpClientBuilder"),
    HttpGet = Java.type("org.apache.http.client.methods.HttpGet"),
    HttpResponse = Java.type("org.apache.http.HttpResponse"),
    HttpEntity = Java.type("org.apache.http.HttpEntity"),
    EntityUtils = Java.type("org.apache.http.util.EntityUtils"),
    HashMap = Java.type("java.util.HashMap");


So, with all those now available in my script, I could begin copying the same implementation that I had.

Once I had my response coming through, I needed to add it to a HashMap, for a Clob datatype (I discovered that SQLcl cuts off values larger than 4000 characters, so it seems a Clob was necessary - especially if the web service response becomes too large).

I took a look at some example code, and saw an example with a blob with `conn.createBlob()`. So I copied this style but used it for a clob instead, as per the following example:

map = new HashMap();
xmlAsClob = conn.createClob();
xmlAsClob.setString(1, respBody);

map.put("xml", xmlAsClob);


This map allows us to use a bind value (xml) with our query using the provided function `util.execute`.

So, for this basic example, I just wanted to log the count that was returned in the XML to run each day. So, for the insert statement, I came up with the following query and implementation:

insertCountQuery =
"insert into org_count (date_run, count)" +
"select " +
"    sysdate" +
"  , orgs.org_count " +
"from " +
"    xmltable( " +
"        '/Result' " +
"        passing xmltype.createxml(:xml) " +
"            columns " +
"                org_count number path '/Result/count' " +
"    ) orgs";

insertCountResult = util.execute(insertCountQuery, map);

if (insertCountResult){
    print("Successfully updated from the web service");
} else {
    print ("An error occurred");
    print ("sqldev.last.err.message.forsqlcode reported:")
    print (ctx.getProperty("sqldev.last.err.message.forsqlcode"));
}


I saved this script as `fetchAndLoad.js`. So when I'm connected to SQLcl, I just need to run the command: `script fetchAndLoad.js`. And voila - our data is sent into your table.

To get this to run daily, it would be a good idea to wrap this in a little shell script and add it to your crontab (or task scheduler if you're on windows).

Take a look at the full example code: https://github.com/tschf/orcl-wssync-sqlcl-demo