Simplest example of monitoring could be go's gc behaviour. In order to do this, we need to collect metrics, which are exposed through go's runtime package. More details about it can be found here: https://golang.org/pkg/runtime/

Collecting metrics from our code

There is already a very good library written in go, which can do this for us, and even more then that. It already has all we need for go's gc metrics, plugins for different backends (grafite, grafana, StatHat etc..), and enables registration of custom metrics collection in a very easy way.
Code is on github: https://github.com/rcrowley/go-metrics
To use it, we need to add imports and register metrics collection, as shown bellow:

package main

import (  
    "fmt"
    "net/http"
    "time"

    "github.com/rcrowley/go-metrics"
    "github.com/vrischmann/go-metrics-influxdb"
)

func main() {

    r := metrics.NewRegistry()
    metrics.RegisterDebugGCStats(r)
    metrics.RegisterRuntimeMemStats(r)

    go metrics.CaptureDebugGCStats(r, time.Second*5)
    go metrics.CaptureRuntimeMemStats(r, time.Second*5)

    go influxdb.InfluxDB(
        r,                       // metrics registry
        time.Second*5,           // interval
        "http://localhost:8086", // the InfluxDB url
        "helloWorld",                 // your InfluxDB database
        "",                  // your InfluxDB user
        "",            // your InfluxDB password
    )

// Rest of the code ...

InfluxDB and Grafana

Above will enable metrics collection and send it to influxDB. Next thing is getting InfluxDB and Grafana. There is a lot of information available on setting up InfluxDB and Grafana, but to keep things simple, we will use docker to run them.

Start InfluxDB with (using default ports 8083 and 8086):

docker run -d -p 8083:8083 -p 8086:8086 tutum/influxdb  

and Grafana with (using port 9090 to access grafana):

docker run -d -p 9090:3000/tcp --name=grafana grafana/grafana  

Create InfluxDB database

Open the browser and go to http://localhost:8083.
Let's create database for our measurements, so type in the following query and hit enter:

CREATE DATABASE "helloWorld"  

Now that we have database created, let's configure datasource in Grafana.

Configuring Grafana Datasource

Open the browser again and go to http://localhost:9090.
It will open login for grafana, default credentials are admin/admin.

In the top left corner click on Grafana logo and select "Data Sources" and then "Add data source", and fill in as shown on the image, clicking "Add" when done. It should show that datasource has been created and is working.

We have now influxDB set up, and Grafana datasource created. We can run our go software and do some monitoring. So run build the code and run it.

Graph the collected metrics

Once we have datasource set up, we can start producing graphs. So create new dashboard by clicking again at top right corner (Grafana logo) and selecting new dashboard. This is where (at least for me), Grafana UI becomes a bit confusing. To create new graph, click on the little rectangle on the left edge of the screen, as shown bellow:

Next we need to select datasource, instead of example provided by Grafana, when it created new graph for us. So change the datasource to our influxdb one called helloWorld:

Then we can edit the query and pick some of our metrics. Example code shown at the start of the text will expose go's runtime metrics, so we can graph something from there. Pick some values for the graph.

Don't forget to change data units on the graph axes, for example if using something from runtime stats, ie Allocated memory, set the Y-axis units to bytes:

Once you tell Grafana to update graph every n-seconds or do manual refresh, data will be drawn on the graph.

Simple enough for a start.
Happy Graphing!