Create API for GCE Metadata

[JustinBeckwith]

There's a great metadata service for getting access to information on the VM. It would be great to have a structured API for this:
https://cloud.google.com/compute/docs/metadata?hl=en

[JustinBeckwith]

For reference, the Go implementation:
https://github.com/GoogleCloudPlatform/gcloud-golang/blob/master/compute/metadata/metadata.go

[dhermes]

We already use the metadata service to auto-detect the current project on GCE. As a first start we could move that functionality into a sub-package.

@tseaver @jgeewax Any preferences on what the subpackage should be named?

[stephenplusplus]

So far, our APIs assume you're talking to the googleapis.com API, regardless of how you auth. We have recently landed support for Resource Manager and are working on Logging, when things get more complex. Specifically, a portion of our API-- creating a Project (RM) or a Sink (Logging)-- requires user auth, as opposed to from a service account. This can quickly confuse our users, since it's up to our docs to explain how, when, & where to use auth and run our code. So this method is another example where we have a method that may or may not work when actually called.

Are there other actions that can be performed only when a user is running on a GCE instance? Possibly enough that could warrant creating a new module that has a dream-suite of utilities for someone running inside GCE?

[JustinBeckwith]

Not that I know of, but it's entirely possible. This will be a common thread though for the compute environment based APIs. For example, there are App Engine APIs we need to build veneers for that dig out App Engine specific information that only work when running inside of App Engine:

https://cloud.google.com/appengine/docs/python/refdocs/google.appengine.api.modules.modules

I think those use standard auth mechanisms though based on this:
https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1beta4/apps.modules/get

[stephenplusplus]

There's a library that looks like it was built to be a GAE suite of tools:

This library will only work inside App Engine Managed VMs, or with the Managed VM SDK.

• https://github.com/GoogleCloudPlatform/appengine-nodejs

It doesn't look like it's been updated recently, so I'm not sure if it has any plans for expansion in sight. However, that seems like the right approach, as it's obvious to the user that their GAE app will benefit from this GAE-only toolkit. That could set the precedent as well for GCE, even if it starts as just a module with one function, "getMetadata". Heck, I'll write it!

// @jgeewax any thoughts?

[JustinBeckwith]

Yeah - we really need to take appengine-nodejs down (we don't want people using that). It's talking to the App Engine v1 APIs with node, which was never really intended.

The App Engine APIs I'm talking about are available via a One Platform API:
https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1beta4/apps.modules/get

[stephenplusplus]

The API seems pretty straightforward, so I wouldn't mind writing a library for it. Just to review, my proposal here is:

1. Write a separately-installable GCE-env-only Node module.

   var env = require('gcloud-compute-engine');
   env.getMetadata(function(err, metadata) {});

2. Write a separately-installable GAE-env-only Node module.

   var env = require('gcloud-appengine');
   env.getApps(function(err, apps) {});
   
   var app = env.app('my-site');
   app.getModules(function(err, modules) {});
   
   var appVersion1 = app.module('1');
   appVersion1.delete(function(err) {});

The whole point being, where confusion is possible, simply narrowing the scope of what a user can expect from a given library we offer.

[JustinBeckwith]

Interesting - why separate these specifically into their own NPM modules? Splitting this up seems kind of confusing, unless we choose to split up the whole module.

Instead, I could see adding something like:

let gcloud = require('gcloud');
gcloud.appEngine.isAvailable(function(err, isAvailable) {
  if (!err && isAvailable) {
    gcloud.appEngine.getModules(function(err, modules) {
      console.log(modules);
    });
  } else {
    console.log('not running in app engine!');
});

That way I can write code to specifically handle the case where I'm not in a particular environment. Same for GCE:

let gcloud = require('gcloud');
gcloud.computeEngine.isAvailable(function(err, isAvailable) {
  if (!err && isAvailable) {
    gcloud.computeEngine.getMetadata(path, function(err, res) {
      console.log(res);
    });
  } else {
    console.log('not running in compute engine!');
  }
});

If isAvailable doesn't sound right, getExecutionEnvironment is another option:

let gcloud = require('gcloud');
gcloud.getExecutionEnvironment(function(err, env) {
  if (!err && env === gcloud.environments.appEngine) {    
      console.log("I'm in app engine!");
  } else {
      console.log('not running in app engine!');
  }
});

[JustinBeckwith]

@jmdobry and @jonparrott interested in your thoughts here too.

[theacodes]

• I don't want to create another appengine specific API, ever. They can just use environment variables for stuff like getting the current module.
• getExecutionEnvironment is the 100% wrong way to do this, I think. You're either on GCP or not. Being on GAE shouldn't matter vs being on GKE or GCE. You're either in GCP or not.
• The metadata API is incredibly useful, and having an API to do things like watch for changes would be extremely useful for users. I don't understand the desire to put it in a separate npm module as it's applicable to all of our hosting platforms. I suppose node.js doesn't have anything like Python's namespace packages.

[stephenplusplus]

why separate these specifically into their own NPM modules?

One reason is simply to shrink the overhead of this repo. The bigger this library grows, it's simply harder to maintain and put out a new release. If we stabilize at 1.0, then catch a breaking change in one of (smallest possible number here) APIs we support, we have to force a major increment upgrade (+1.0) on all of our users to have them benefit from smaller releases with features and fixes.

That aside, I prefer this split to avoid confusion. I think a user should be able to install a library and have everything work. Then, if they need something more specific to their application, they can install another library. I mentioned earlier the Logging and Resource Manager API muddying things up a bit earlier, but in those cases, we at least give the user the majority of the API regardless of what environment they're in.

With these extra APIs for App and Compute Engine, I think the separation will make it more clear for a user to understand what to use given their environment. Also, it will help users who may only need the GAE/GCE functionality by offering that library as separately installable.

[stephenplusplus]

I don't want to create another appengine specific API, ever.

Why is that?

You're either in GCP or not.

Basically, the user shouldn't need this behavior because they're the ones deploying their app, so they should know where it's at?

[theacodes]

Why is that?

Because from my point of view, App Engine is not a special snowflake in our hosting options.

Basically, the user shouldn't need this behavior because they're the ones deploying their app, so they should know where it's at?

I'm just saying that at most, we should have

var metadata = require('gcloud').metadata();

metadata.isAvailable(function(err) { ... });

Ideally, there should be little need to distinguish between GAE, GCE, and GKE. Even if that is desired, the metadata API is the wrong place to do it.

[stephenplusplus]

Ideally, there should be little need to distinguish between GAE, GCE, and GKE

That makes sense to me. The idea of being able to stop/delete a module programmatically sounded cool, though.

But, if all we want is a "try to get metadata" method, this is fine with me?:

var gcloud = require('gcloud');

gcloud.getMetadata(function(err, metadata) {
  if (/*not in a place we can get metadata*/) {
    err.message = 'Not in a place we can get metadata.';
  }
});

[theacodes]

FYI, I believe the ideal API surface for the metadata server should look something like this:

var metadata = require('gcloud').metadata();

// get instance tags.
metadata.instance.tags(function(err, tags) {
  console.log(tags);
  // ['one', 'two', 'blue', 'shoe']
});

// Get project-level custom metadata.
metadata.project.custom(function(err, kv) {
  console.log(kv);
  // {color: "red", active: "yes"}
});

// watch for changes to the instance tags.
metadata.instance.tags.watch(function(err, tags) {
   //...
});