Problem: I want to get my Google Analytics data into my own “backend system”
Solution: an example of how to download google analytics data without any realtime user intervention required (no Auth windows etc), using a Google Cloud service account and Go.
there are a number of pre-requisite steps that must be completed before we can get Go-ing. lets get the non-Go steps done first.
non-Go pre-requisite steps:
- ACCOUNT: you’ve got to have a Google Analytics account & property with valid, query-able data. additionally, because you’ll be querying the Core Reporting API using the Google APIs Client Library for Go, you’ll need a Google Cloud application to do it.
- PROJECT: create a Google Cloud project. to create one of those, you need to have an account on the Google Cloud Console, and have created a Project in that account.
- ANALYTICS API: turn on the Analytics API for your project. enable the Analytics API for the Project. find “Analytics” in the APIs & Auth > API’s submenu on the left, and switch it to ON.
- APP: create a Google Cloud Application. it’s time to create an Application! click “Register App” in the Registered Apps submenu. give your app a name (more of a description, really) and make it of type “Web Application”.
- CREDENTIALS. your headless application needs credentials (an email address & public/private key pair) so it can be authorized to query the Core Reporting API.create them by expanding “Certificates” in the Registered Apps > YourAppName. you’ll see that there is no public key fingerprint, and email address says “Not generated yet”.
click “Generate Certificate”. you’ll be prompted to download the private key. save this and keep it safe (you can only download it once, ever).
it will also tell you the private key’s password; make a note of this, as you’ll need it when converting the .p12 key you just downloaded into a .pem key (which is what Google’s OAuth requires for service accounts) later.
- JSON: while making a note of your freshly generated service account email address (it looks like
8961649gibberishkjxcjhbsdv@developer.gserviceaccount.com), you should download the JSON file (this is your application’s “client secrets” file). you will set the variable gaServiceAcctSecretsFile to the location of the file later. - PEM KEY: we need to convert the .p12 into a .pem key. ok, let’s convert the .p12 key into a .pem key, because that’s the key format required to sign the authentication requests.
$ openssl pkcs12 -in privatekeyfilename.p12 -nodes -nocerts > privatekeyfilename.pemyou’ll be asked for the Import password. enter the private key’s password you were given above. now you’ve got the PEM key! you will be setting the variable gaServiceAcctPEMKey to the location of this file later.
- ANALYTICS PERMISSIONS: now you have to grant the application appropriate permissions to query your Analytics account/property. within Google Analytics, in the Admin section specific to the account/property you want to query, give “Read & Analyze” permissions to the @developer.gserviceaccount.com address you made a note of earlier.
- ANALYTICS TABLE ID: okay, just one more step! the Core Reporting API requires us to identify the analytics dataset we want to query by using a TableID, not (as you might think), an analytics tracking ID.we can find the TableID in the Google Analytics Query Explorer – which by the way, is fantastic.
login there, navigate to the specific account/property/profile, and you will see the TableID in the “ids” box. it will look something like:
ga:12345678
make a note of it, you setting the variable gaTableID to this value later.
w00t! that’s all of the pre-requisite steps done & dusted. let’s look at some Go code (finally).
Go pre-requisite steps:
in order to use the Go Google API, it must be installed. to install all of the Go Google API’s (Drive, BigQuery, Calendar, etc), you can just do:
$ go get golang.org/x/oauth2
$ go get google.golang.org/api/analytics/v3
once you’ve ensured that you’ve got the above packages safely import-able in your $GOPATH, you’re good to go.
no more pre-requisite steps. let’s talk code.
head on over to htps://github.com/rickt/analyticsdumper. full instructions and example run are there! but here’s the sauce since that’s why we’re all here 🙂
UPDATED source:
| package main | |
| import ( | |
| "fmt" | |
| "golang.org/x/oauth2" | |
| "golang.org/x/oauth2/jwt" | |
| "google.golang.org/api/analytics/v3" | |
| "io/ioutil" | |
| "log" | |
| "time" | |
| ) | |
| // constants | |
| const ( | |
| datelayout string = "2006-01-02" // date format that Core Reporting API requires | |
| ) | |
| // globals that you DON'T need to change | |
| var ( | |
| enddate string = time.Now().Format(datelayout) // set end query date to today | |
| startdate string = time.Now().Add(time.Hour * 24 * –1).Format(datelayout) // set start query date to yesterday | |
| metric string = "ga:pageviews" // GA metric that we want | |
| tokenurl string = "https://accounts.google.com/o/oauth2/token" // (json:"token_uri") Google oauth2 Token URL | |
| ) | |
| // globals that you DO need to change | |
| // populate these with values from the JSON secretsfile obtained from the Google Cloud Console specific to your app) | |
| // example secretsfile JSON: | |
| // { | |
| // "web": { | |
| // "auth_uri": "https://accounts.google.com/o/oauth2/auth", | |
| // "token_uri": "https://accounts.google.com/o/oauth2/token", | |
| // "client_email": "blahblahblahblah@developer.gserviceaccount.com", | |
| // "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/blahblahblahblah@developer.gserviceaccount.com", | |
| // "client_id": "blahblahblahblah.apps.googleusercontent.com", | |
| // "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs" | |
| // } | |
| // } | |
| var ( | |
| // CHANGE THESE!!! | |
| gaServiceAcctEmail string = "blahblahblahblah@developer.gserviceaccount.com" // (json:"client_email") email address of registered application | |
| gaServiceAcctPEMKey string = "./analyticsdumper.pem" // full path to private key file (PEM format) of your application from Google Cloud Console | |
| gaTableID string = "ga:NNNNNNNN" // namespaced profile (table) ID of your analytics account/property/profile | |
| ) | |
| // func: main() | |
| // the main function. | |
| func main() { | |
| // load up the registered applications private key | |
| pk, err := ioutil.ReadFile(gaServiceAcctPEMKey) | |
| if err != nil { | |
| log.Fatal("Error reading GA Service Account PEM key -", err) | |
| } | |
| // create a jwt.Config that we will subsequently use for our authenticated client/transport | |
| // relevant docs for all the oauth2 & json web token stuff at https://godoc.org/golang.org/x/oauth2 & https://godoc.org/golang.org/x/oauth2/jwt | |
| jwtc := jwt.Config{ | |
| Email: gaServiceAcctEmail, | |
| PrivateKey: pk, | |
| Scopes: []string{analytics.AnalyticsReadonlyScope}, | |
| TokenURL: tokenurl, | |
| } | |
| // create our authenticated http client using the jwt.Config we just created | |
| clt := jwtc.Client(oauth2.NoContext) | |
| // create a new analytics service by passing in the authenticated http client | |
| as, err := analytics.New(clt) | |
| if err != nil { | |
| log.Fatal("Error creating Analytics Service at analytics.New() -", err) | |
| } | |
| // create a new analytics data service by passing in the analytics service we just created | |
| // relevant docs for all the analytics stuff at https://godoc.org/google.golang.org/api/analytics/v3 | |
| ads := analytics.NewDataGaService(as) | |
| // w00t! now we're talking to the core reporting API. the hard stuff is over, lets setup a simple query. | |
| // setup the query, call the Analytics API via our analytics data service's Get func with the table ID, dates & metric variables | |
| gasetup := ads.Get(gaTableID, startdate, enddate, metric) | |
| // send the query to the API, get a big fat gaData back. | |
| gadata, err := gasetup.Do() | |
| if err != nil { | |
| log.Fatal("API error at gasetup.Do() -", err) | |
| } | |
| // print out some nice things | |
| fmt.Printf("%s pageviews for %s (%s) from %s to %s.\n", gadata.Rows[0], gadata.ProfileInfo.ProfileName, gadata.ProfileInfo.WebPropertyId, startdate, enddate) | |
| return | |
| } |
