Sending custom application metrics to the Stackdriver system

Stackdriver supports ingesting, displaying and alerting on custom metrics sent from your application. To view your metrics after you have sent them, head to app.stackdriver.com/custom-metrics.

To send custom metrics, take the following steps. 

  1. Create a new API key in your Stackdriver Account Settings
  2. Submit Custom Metrics with the API Key
(Code snippets in this section are written in Python.)

To push custom metrics to Stackdriver, you will POST a message to the gateway and include the API key in the HTTP header.

Each data point must contain three fields: name, value, and collected_at.  
  • ‘name’ is the name of your metric; you can define this using any taxonomy that helps you organize your data. 
  • ‘value’ is the measurement to record for the data point, and should be passed as an integer or float/decimal.  
  • ‘collected_at’ is the time that the data point was collected, in Unix epoch time, and should be passed as an integer.  If this value is over an hour before the current time, the measurement will be rejected.


By default, custom metrics are not associated with any instance and can be found under the Custom resource type when creating charts or alerting policies.  


If your custom metric is instead related to a particular instance, you can add an optional instance field to your message with a value of the instance id for your instance (eg, i-1234b3f3). These metrics then show up under the Instance resource. This makes it easier to have one metric name shared across a number of instances to include on a single graph or for alerting.

Example data point

data_point = {
   ‘name’: ‘my_custom_metric’,
   ‘value’: 32,  #Note: Stackdriver rejects when the collected_at value is older than 2 hours.
   ‘collected_at’: 1325394000 }

The data point is submitted to the gateway as part of a gateway message.  The gateway message includes the timestamp the message is created or submitted, your customer identifier, and the protocol version for the message.  The timestamp is used to disambiguate messages.  Messages with the same timestamp are assumed to be the same (see below).  The ‘timestamp’ field must be an integer.

The protocol version defines the schema of the message. The current protocol version (and that which is described in this document) is version 1.

Example gateway message

    sd_post = {
        'timestamp': int(time.time()),
        'proto_version': 1,
        'data': data_point,
        }

The final message preparation is defining the HTTP headers.  The gateway protocol requires that messages be JSON-encoded.  The API key is also included in the header.

Example headers

 headers = {
        'content-type': 'application/json',
        'x-stackdriver-apikey': args.apikey
        }
Submit the measurement to Stackdriver by POSTing it to the gateway at https://custom-gateway.stackdriver.com/v1/custom.  In the code fragment below, we use the Python “requests” module for HTTP messaging; it can be installed via ‘pip install requests’ or ‘easy_install requests’.

Example http request

 import requests
 resp = requests.post('https://custom-gateway.stackdriver.com/v1/custom',
                       data=json.dumps(sd_post),
                       headers=headers)

Putting it all together


A complete working example that combines all the pieces we have discussed in shown below.

Note: We will not accept metrics sent more frequently than once per minute. If you need to send metrics at a higher frequency, please read our article on sending aggregated metrics with statsd.

    import requests
    import json
    import time
    def submit_custom_metric():
        data_point = {
            'name': 'my_custom_metric',
            'value': 3.14159, #Note: Stackdriver rejects when the collected_at value is older than 2 hours.
            'collected_at': 1325394000,
            }
        gateway_msg = {
            'timestamp': int(time.time()),
            'proto_version': 1,
            'data': data_point,
            }
        headers = {
            'content-type': 'application/json',
            'x-stackdriver-apikey': '4VP31M3RJ5JWN29E1LMSE0'
            }
        resp = requests.post( 'https://custom-gateway.stackdriver.com/v1/custom', data=json.dumps(gateway_msg), headers=headers) if not resp.ok: print resp.content assert resp.ok, 'Failed to submit custom metric.' else: print "Published."
    if __name__ == '__main__':
        submit_custom_metric()

Multiple Data Points in a Message

The custom metrics API also supports sending a list of data points in the data portion of the message payload.  Otherwise, everything else from the above is exactly the same.  An example of such a message would be to replace gateway_msg in the last example with the following
gateway_msg = {
        'timestamp': int(time.time()),
        'proto_version': 1,
        'data': [ { 'name': 'my_custom_metric1', 'value': 32, #Note: Stackdriver rejects when the collected_at value is older than 2 hours. 'collected_at': 1325394000, }, { 'name': 'my_custom_metric2', 'value': 37,  #Note: Stackdriver rejects when the collected_at value is older than 2 hours. 'collected_at': 1325394000, } ] }

Note that each point within the data element must have its own (not necessarily unique) name, value, and collected_at. Keep in mind that the policy regarding the age of collected_at still applies.

Libraries

A few libraries exist for sending custom metrics to Stackdriver.  Note that these are contributed by other users and not by Stackdriver

Removing Custom Metrics


If you accidentally send incorrect metric keys, you can remove them from being seen in Stackdriver by following the instructions here.  It's worth noting that this removes the key and not the underlying data so if you begin sending a metric with the same name again, the old data will still be visible until it expires naturally with the rest of your metric data.

Feedback and Knowledge Base