i decided to make my G Suite “mdmtool” a bit more robust and release-worthy, instead of just proof of concept code.
(link: https://github.com/rickt/mdmtool) github.com/rickt/mdmtool.
enjoy!
i decided to make my G Suite “mdmtool” a bit more robust and release-worthy, instead of just proof of concept code.
(link: https://github.com/rickt/mdmtool) github.com/rickt/mdmtool.
enjoy!
Our desktop support & G Suite admin folks needed a simple, fast command-line tool to query basic info about our company’s mobile devices (which are all managed using G Suite’s built-in MDM).
So I wrote one.
Since this tool needs to be run via command-line, it can’t use any interactive or browser-based authentication, so we need to use a service account for authentication.
Pre-requisites (GCP & G Suite):
client_id
. Make a note of it!client_id
of your service account for the necessary API scopes. In the Admin Console for your G Suite domain (Admin Console –> Security –> Advanced Settings –> Authentication –> Manage API Client Access), add your client_id
in the “Client Name” box, and add https://www.googleapis.com/auth/admin.directory.device.mobile.readonlyin the “One or more API scopes” box
Pre-requisites (Go)
You’ll need to “go get” a few packages:
go get -u golang.org/x/oauth2/google
go get -u google.golang.org/api/admin/directory/v1
go get -u github.com/dustin/go-humanize
Pre-requisites (Environment Variables)
Because it’s never good to store runtime configuration within code, you’ll notice that the code references several environment variables. Setup them up to suit your preference but something like this will do:
export GSUITE_COMPANYID="A01234567" export SVC_ACCOUNT_CREDS_JSON="/home/rickt/dev/adminsdk/golang/creds.json export GSUITE_ADMINUSER="foo@bar.com"
And finally, the code
Gist URL: https://gist.github.com/rickt/199ca2be87522496e83de77bd5cd7db2
i wanted to try out the automatic loading of CSV data into Bigquery, specifically using a Cloud Function that would automatically run whenever a new CSV file was uploaded into a Google Cloud Storage bucket.
it worked like a champ. here’s what i did to PoC:
$ head -3 testdata.csv id,first_name,last_name,email,gender,ip_address 1,Andres,Berthot,aberthot0@pbs.org,Male,104.204.241.0 2,Iris,Zwicker,izwicker1@icq.com,Female,61.87.224.4
$ gsutil mb gs://csvtestbucket
$ pip3 install google-cloud-bigquery --upgrade
$ bq mk --dataset rickts-dev-project:csvtestdataset
$ bq mk -t csvtestdataset.csvtable \ id:INTEGER,first_name:STRING,last_name:STRING,email:STRING,gender:STRING,ip_address:STRING
BUCKET: csvtestbucket DATASET: csvtestdataset TABLE: csvtable VERSION: v14
google-cloud google-cloud-bigquery
*csv *yaml
$ ls env.yaml main.py requirements.txt testdata.csv
$ gcloud beta functions deploy csv_loader \ --runtime=python37 \ --trigger-resource=gs://csvtestbucket \ --trigger-event=google.storage.object.finalize \ --entry-point=csv_loader \ --env-vars-file=env.yaml
$ gsutil cp testdata.csv gs://csvtestbucket/ Copying file://testdata.csv [Content-Type=text/csv]... - [1 files][ 60.4 KiB/ 60.4 KiB] Operation completed over 1 objects/60.4 KiB.
$ gcloud functions logs read [ ... snipped for brevity ... ] D csv_loader 274732139359754 2018-10-22 20:48:27.852 Function execution started I csv_loader 274732139359754 2018-10-22 20:48:28.492 Starting job 9ca2f39c-539f-454d-aa8e-3299bc9f7287 I csv_loader 274732139359754 2018-10-22 20:48:28.492 Function=csv_loader, Version=v14 I csv_loader 274732139359754 2018-10-22 20:48:28.492 File: testdata2.csv I csv_loader 274732139359754 2018-10-22 20:48:31.022 Job finished. I csv_loader 274732139359754 2018-10-22 20:48:31.136 Loaded 1000 rows. D csv_loader 274732139359754 2018-10-22 20:48:31.139 Function execution took 3288 ms, finished with status: 'ok'
looks like the function ran as expected!
$ bq show csvtestdataset.csvtable Table rickts-dev-project:csvtestdataset.csvtable Last modified Schema Total Rows Total Bytes Expiration Time Partitioning Labels ----------------- ----------------------- ------------ ------------- ------------ ------------------- -------- 22 Oct 13:48:29 |- id: integer 1000 70950 |- first_name: string |- last_name: string |- email: string |- gender: string |- ip_address: string
great! there are now 1000 rows. looking good.
$ egrep '^[1,2,3],' testdata.csv 1,Andres,Berthot,aberthot0@pbs.org,Male,104.204.241.0 2,Iris,Zwicker,izwicker1@icq.com,Female,61.87.224.4 3,Aime,Gladdis,agladdis2@hugedomains.com,Female,29.55.250.191
with the first 3 rows of the bigquery table
$ bq query 'select * from csvtestdataset.csvtable \ where id IN (1,2,3)' Waiting on bqjob_r6a3239576845ac4d_000001669d987208_1 ... (0s) Current status: DONE +----+------------+-----------+---------------------------+--------+---------------+ | id | first_name | last_name | email | gender | ip_address | +----+------------+-----------+---------------------------+--------+---------------+ | 1 | Andres | Berthot | aberthot0@pbs.org | Male | 104.204.241.0 | | 2 | Iris | Zwicker | izwicker1@icq.com | Female | 61.87.224.4 | | 3 | Aime | Gladdis | agladdis2@hugedomains.com | Female | 29.55.250.191 | +----+------------+-----------+---------------------------+--------+---------------+
and whaddyaknow, they match! w00t!
proof of concept: complete!
conclusion: cloud functions are pretty great.
lets say you have some servers in a cluster serving vhost foo.com and you want to put all the access logs from all the webservers for that vhost into Bigquery so you can perform analyses, or you just want all the access logs in one place.
in addition to having the raw weblog data, you also want to keep track of which webserver the hits were served by, and what the vhost (Host header) was.
so, foreach()
server, we will install fluentd
, configure it to tail the nginx access log, and upload everything to Bigquery for us.
it worked like a champ. here’s what i did to PoC:
fluentd
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh
$ bq mk --dataset rickts-dev-project:nginxweblogs Dataset 'rickts-dev-project:nginxweblogs' successfully created.
[ { "name": "agent", "type": "STRING" }, { "name": "code", "type": "STRING" }, { "name": "host", "type": "STRING" }, { "name": "method", "type": "STRING" }, { "name": "path", "type": "STRING" }, { "name": "referer", "type": "STRING" }, { "name": "size", "type": "INTEGER" }, { "name": "user", "type": "STRING" }, { "name": "time", "type": "INTEGER" }, { "name": "hostname", "type": "STRING" }, { "name": "vhost", "type": "STRING" } ]
$ bq mk -t nginxweblogs.nginxweblogtable schema.json Table 'rickts-dev-project:nginxweblogs.nginxweblogtable' successfully created.
$ sudo /usr/sbin/td-agent-gem install fluent-plugin-bigquery --no-ri --no-rdoc -V
fluentd
to read the nginx access log for this vhost and upload to Bigquery (while also adding the server hostname and vhost name) by creating an /etc/td-agent/td-agent.conf
similar to this: https://gist.github.com/rickt/641e086d37ff7453b7ea202dc4266aa5 (unfortunately WordPress won’t render it properly, sorry)
You’ll note we are using the record_transformer
fluentd filter plugin to transform the access log entries with the webserver hostname and webserver virtualhost name before injection into Bigquery.
fluentd
runs as (td-agent by default
) has read access to your nginx access logs, start (or restart) fluentd
$ sudo systemctl start td-agent.service
$ hostname hqvm $ curl http://localhost/index.html?text=helloworld you sent: "helloworld"
$ bq query 'SELECT * FROM nginxweblogs.nginxweblogtable WHERE path = "/index.html?text=helloworld"' +-------------+------+------+--------+-----------------------------+---------+------+------+------+----------+--------------------------+ | agent | code | host | method | path | referer | size | user | time | hostname | vhost | +-------------+------+------+--------+-----------------------------+---------+------+------+------+----------+--------------------------+ | curl/7.47.0 | 200 | ::1 | GET | /index.html?text=helloworld | - | 14 | - | NULL | hqvm | rickts-dev-box.fix8r.com | +-------------+------+------+--------+-----------------------------+---------+------+------+------+----------+--------------------------+
proof of concept: complete!
conclusion: pushing your web access logs into Bigquery is extremely easy, not to mention, a smart thing to do.
the benefits exponentially increase as your server + vhost count increases. try consolidating, compressing and analyzing logs from N+ servers using months of data in-house and you’ll see the benefits of Bigquery right away.
enjoy!
stable release of Slack Translator Bot.
http://github.com/rickt/slack-translator-bot
what is Slack Translator Bot? the [as-is demo] code gets you get a couple of Slack /slash
commands that let you translate from English to Japanese, and vice-versa.
below screenshot shows example response to a Slack user wanting to translate “the rain in spain falls mainly on the plane” by typing:
within slack:
TL;DR/HOW-TO
stable release of Slack Team Directory Bot.
http://github.com/rickt/slack-team-directory-bot
what is Slack Team Directory Bot? you get a Slack /slash
command that lets you search your Slack Team Directory quick as a flash.
below screenshot shows example response to a Slack trying to find someone in your accounting department by typing:
within slack:
TL;DR/HOW-TO
http://gist-it.appspot.com/http://github.com/rickt/slack-team-directory-bot/blob/master/slackteamdirectorybot.go
rickt/slack-team-directory-bot
i’ve updated my example Golang code that authenticates with the Core Reporting API using service account OAuth2 (two-legged) authentication to use the newly updated golang.org/x/oauth2 library.
my previous post & full explanation of service account pre-reqs/setup:
http://code.rickt.org/post/142452087425/how-to-download-google-analytics-data-with-golang
full code:
have fun!
drop me a line or say hello on twitter if any questions.
stable release of my modified-for Google Appengine fork of https://github.com/bluele/slack. i have this working in production on several Appengine-hosted /slash commands & bots.
http://github.com/rickt/slack-appengine
the TL;DR on my modifications:
https://github.com/rickt/slack-appengine
stable release of Slack Timebot.
https://github.com/rickt/timebot-simple
what is Slack Timebot? at work, i very often have to know what time it is in the following regions:
so i wrote a mini go backend app that i threw into a free Google Appengine app and so now i can get the time instantly by using any of these new Slack /slash commands:
TL;DR/HOW-TO
http://gist-it.appspot.com/http://github.com/rickt/timebot-simple/blob/master/timebot-simple.go
Problem: I want to get my Google Analytics data into my own “backend system”
Solution: an example of how to download google analytics data without any realtime user intervention required (no Auth windows etc), using a Google Cloud service account and Go.
there are a number of pre-requisite steps that must be completed before we can get Go-ing. lets get the non-Go steps done first.
click “Generate Certificate”. you’ll be prompted to download the private key. save this and keep it safe (you can only download it once, ever).
it will also tell you the private key’s password; make a note of this, as you’ll need it when converting the .p12 key you just downloaded into a .pem key (which is what Google’s OAuth requires for service accounts) later.
8961649gibberishkjxcjhbsdv@developer.gserviceaccount.com
), you should download the JSON file (this is your application’s “client secrets” file). you will set the variable gaServiceAcctSecretsFile to the location of the file later.$ openssl pkcs12 -in privatekeyfilename.p12 -nodes -nocerts > privatekeyfilename.pem
you’ll be asked for the Import password. enter the private key’s password you were given above. now you’ve got the PEM key! you will be setting the variable gaServiceAcctPEMKey to the location of this file later.
login there, navigate to the specific account/property/profile, and you will see the TableID in the “ids” box. it will look something like:
ga:12345678
make a note of it, you setting the variable gaTableID to this value later.
w00t! that’s all of the pre-requisite steps done & dusted. let’s look at some Go code (finally).
in order to use the Go Google API, it must be installed. to install all of the Go Google API’s (Drive, BigQuery, Calendar, etc), you can just do:
$ go get golang.org/x/oauth2
$ go get google.golang.org/api/analytics/v3
once you’ve ensured that you’ve got the above packages safely import-able in your $GOPATH, you’re good to go.
no more pre-requisite steps. let’s talk code.
head on over to htps://github.com/rickt/analyticsdumper. full instructions and example run are there! but here’s the sauce since that’s why we’re all here 🙂
UPDATED source:
package main | |
import ( | |
"fmt" | |
"golang.org/x/oauth2" | |
"golang.org/x/oauth2/jwt" | |
"google.golang.org/api/analytics/v3" | |
"io/ioutil" | |
"log" | |
"time" | |
) | |
// constants | |
const ( | |
datelayout string = "2006-01-02" // date format that Core Reporting API requires | |
) | |
// globals that you DON'T need to change | |
var ( | |
enddate string = time.Now().Format(datelayout) // set end query date to today | |
startdate string = time.Now().Add(time.Hour * 24 * –1).Format(datelayout) // set start query date to yesterday | |
metric string = "ga:pageviews" // GA metric that we want | |
tokenurl string = "https://accounts.google.com/o/oauth2/token" // (json:"token_uri") Google oauth2 Token URL | |
) | |
// globals that you DO need to change | |
// populate these with values from the JSON secretsfile obtained from the Google Cloud Console specific to your app) | |
// example secretsfile JSON: | |
// { | |
// "web": { | |
// "auth_uri": "https://accounts.google.com/o/oauth2/auth", | |
// "token_uri": "https://accounts.google.com/o/oauth2/token", | |
// "client_email": "blahblahblahblah@developer.gserviceaccount.com", | |
// "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/blahblahblahblah@developer.gserviceaccount.com", | |
// "client_id": "blahblahblahblah.apps.googleusercontent.com", | |
// "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs" | |
// } | |
// } | |
var ( | |
// CHANGE THESE!!! | |
gaServiceAcctEmail string = "blahblahblahblah@developer.gserviceaccount.com" // (json:"client_email") email address of registered application | |
gaServiceAcctPEMKey string = "./analyticsdumper.pem" // full path to private key file (PEM format) of your application from Google Cloud Console | |
gaTableID string = "ga:NNNNNNNN" // namespaced profile (table) ID of your analytics account/property/profile | |
) | |
// func: main() | |
// the main function. | |
func main() { | |
// load up the registered applications private key | |
pk, err := ioutil.ReadFile(gaServiceAcctPEMKey) | |
if err != nil { | |
log.Fatal("Error reading GA Service Account PEM key -", err) | |
} | |
// create a jwt.Config that we will subsequently use for our authenticated client/transport | |
// relevant docs for all the oauth2 & json web token stuff at https://godoc.org/golang.org/x/oauth2 & https://godoc.org/golang.org/x/oauth2/jwt | |
jwtc := jwt.Config{ | |
Email: gaServiceAcctEmail, | |
PrivateKey: pk, | |
Scopes: []string{analytics.AnalyticsReadonlyScope}, | |
TokenURL: tokenurl, | |
} | |
// create our authenticated http client using the jwt.Config we just created | |
clt := jwtc.Client(oauth2.NoContext) | |
// create a new analytics service by passing in the authenticated http client | |
as, err := analytics.New(clt) | |
if err != nil { | |
log.Fatal("Error creating Analytics Service at analytics.New() -", err) | |
} | |
// create a new analytics data service by passing in the analytics service we just created | |
// relevant docs for all the analytics stuff at https://godoc.org/google.golang.org/api/analytics/v3 | |
ads := analytics.NewDataGaService(as) | |
// w00t! now we're talking to the core reporting API. the hard stuff is over, lets setup a simple query. | |
// setup the query, call the Analytics API via our analytics data service's Get func with the table ID, dates & metric variables | |
gasetup := ads.Get(gaTableID, startdate, enddate, metric) | |
// send the query to the API, get a big fat gaData back. | |
gadata, err := gasetup.Do() | |
if err != nil { | |
log.Fatal("API error at gasetup.Do() -", err) | |
} | |
// print out some nice things | |
fmt.Printf("%s pageviews for %s (%s) from %s to %s.\n", gadata.Rows[0], gadata.ProfileInfo.ProfileName, gadata.ProfileInfo.WebPropertyId, startdate, enddate) | |
return | |
} |