a slack mini “sniffer”; connects to slack and (assuming a valid token), outputs the messages received over the slack websocket
https://github.com/rickt/golang-slack-tools/blob/master/slackminisniffer.go
a slack mini “sniffer”; connects to slack and (assuming a valid token), outputs the messages received over the slack websocket
https://github.com/rickt/golang-slack-tools/blob/master/slackminisniffer.go
stable release of Slack Timebot.
https://github.com/rickt/timebot-simple
what is Slack Timebot? at work, i very often have to know what time it is in the following regions:
so i wrote a mini go backend app that i threw into a free Google Appengine app and so now i can get the time instantly by using any of these new Slack /slash commands:
TL;DR/HOW-TO
http://gist-it.appspot.com/http://github.com/rickt/timebot-simple/blob/master/timebot-simple.go
[i’d not seen this well-documented on the interwebs so here it is for posterity’s sakes]
problem: you have a “desktop” and a “mobile” site. they’re completely separate infra: you have two separate docroots, two separate vhosts. both sites use numeric IDs to target specific “product pages” , but because marketing departments exist and your mobile site was recently Angular-ized, your desktop and mobile sites have different URL standards and different product ID’s. to make your day even better, you’ve just been told that mobile users who hit desktop-site product URLs need to get redirected to the matching product URL on the mobile site.
the relevant URL schemas:
desktop site:
http://www.foo.com/productpages/NNNNNNN_NNN/index.html
(where N is 0-9)
mobile site:
http://m.foo.com/app/product/?p=NNNNN
(where N is 0-9)
you’re choking right now because those ID differences are just obnoxious and there’s no obvious relationship between the 6_3 digit ID’s (desktop) and 5 digit ID’s (mobile), but it’s going to be okay because your DB guys can give you a table dump with the desktop –> mobile product IDs. ok. so let’s talk specifics. your site has a product, a very thin and barely-purchased pamphlet titled “Great British Sportscars of the 1980s”. the pamphlet’s URLs:
desktop site :
/productpages/672019_029/index.html
mobile site:
/app/product/?p=40083
ok no problem right? useragent inspection in nginx is a cinch, as are nginx rewrites. you also know about nginx maps and how you very easily use them in an
"$OLDURL $NEWURL;"
way. job done, lets go home early, right?
well, all of this is true, but nginx (of course) doesn’t make it terribly obvious how you might combine useragent-based rewrites AND transforming specific ID’s from one schema another. in general, we want to rewrite that mobile user to a URL that nginx has in a map, and then that subsequent map match will send the user onto the correct mobile URL (with the correct mobile product ID) on your mobile site.
the first thing to do is to get the desktop and mobile product ID’s in a map so that nginx can do the desktop –> mobile ID transform for you. how you massage or get your data in this form is entirely up to your infrastructure and imagination, but you want to end up with a file like this:
/_mobile/productpages/672019_029/ http://m.foo.com/app/product/?p=40083;
/_mobile/productpages/562174_334/ http://m.foo.com/app/product/?p=10834;
/_mobile/productpages/455383_931/ http://m.foo.com/app/product/?p=90211;
/_mobile/productpages/369410_365/ http://m.foo.com/app/product/?p=16388;
the format (basically) is: a slightly different version of your desktop site ID’s + URI on the left, mobile site ID’s + full URL on the right. so, assuming your txt file is
/etc/nginx/vhostconf.d/www.foo.com/productids.map
, within the nginx server { }
block of your www.foo.com vhost, define the map as per:
map $uri $productid_map {
include /etc/nginx/vhostconf.d/www.foo.com/productids.map;
}
this tells nginx to setup a map, using the contents of your .map file for data. depending on the size of your map(s) you may have to increase the amount of memory (check your hash_bucket_size, map_hash_max_size, map_hash_bucket_size variable values) that nginx allocates for maps & such.
ok, so the map is setup, now we create a rewrite rule to use it. first we’ll create a variable and setup a basic mobile useragent regex match:
set $typeof_request N; if ($http_user_agent ~ "(iPod|iPad|iPhone|BlackBerry|Android|HTC|Motorola)") { set $typeof_request "MOBILE"; }
if the useragent of the request matches our basic check above, the content of the $typeof_request variable is set to MOBILE. now we need to check the URI of the request to see if it matches our desktop product ID URL schema. if it does, we’ll append “_DESKREQ” to $typeof_request.
if ($request_uri ~* ^/productpages/(dddddd_ddd)/.*$) { set $original_productid $1; set $typeof_request "${typeof_request}_DESKREQ"; }
the idea here is that if we get a mobile request for a desktop URI, we want the variable $typeof_request to have the value “MOBILE_DESKREQ”. why? so we can rewrite that mobile request:
if ($typeof_request = MOBILE_DESKREQ) { rewrite ^ $scheme://$host/_mobile/productpages/${original_productid}/; break; }
this rewrite will only occur if the request came from a mobile user and the request was for a specific syntax of URL. lets do an example. assume that someone on a mobile device requests
http://www.foo.com/productpages/672019_029/index.html
first, the $typeof_request variable would be set to MOBILE because of the $http_user_agent check matching their mobile browser useragent string. second, since the URL of the request matches our desktop product ID URL schema, _DESKREQ would be appended to $typeof_request making its value MOBILE_DESKREQ.
and so when the final $typeof_request check is done, $typeof_request is indeed set to MOBILE_DESKREQ and the URL would be rewritten to
http://www.foo.com/_mobile/productpages/672019_029/.
at this point you’re laughing because you already have an nginx map configured to look for strings like
/_mobile/productpages/672019_029/
for the express purpose of easily “mapping” them to strings like
http://m.foo.com/app/product/?p=40083.
all the pieces we need are in place, now we just ask nginx to 301 redirect to the appropriate mobile URL if the requested URI matches any of the slightly modified desktop URIs in our map:
if ($productid_map) { return 301 $productid_map; }
that’s it! a quick overview of the process:
any questions, feel free to say hi on twitter, or drop me a line.
-RMT
i slapped some code together to spit out the currently-connected players on an Arma 3 server.
package main | |
import ( | |
"flag" | |
"fmt" | |
steam "github.com/kidoman/go-steam" | |
"sort" | |
) | |
var addresses = []string{ | |
"public.2-75thrangers.com:2303", | |
} | |
type SteamPlayers steam.PlayersInfoResponse | |
// implement the sort interface on a steam.PlayersInfoResponse-like type | |
// we can't add the sort interface to steam.PlayersInfoResponse because we're not part of the go-steam package, and | |
// so have to use SteamPlayers, which is a new type we created of type steam.PlayersInfoResponse for sort() purposes. | |
// len | |
func (d SteamPlayers) Len() int { | |
return len(d.Players) | |
} | |
// swap | |
func (d SteamPlayers) Swap(i, j int) { | |
d.Players[i], d.Players[j] = d.Players[j], d.Players[i] | |
} | |
// less | |
func (d SteamPlayers) Less(i, j int) bool { | |
return d.Players[i].Score < d.Players[j].Score | |
} | |
func main() { | |
// not really needed because i got lazy and put the IP in a slice as a global var <for shame> | |
flag.Parse() | |
// range through our list of IPs | |
for _, addr := range addresses { | |
// connect to the server | |
server, err := steam.Connect(addr) | |
if err != nil { | |
panic(err) | |
} | |
// save for later | |
defer server.Close() | |
// ping the server. if no ping, something is wrong | |
ping, err := server.Ping() | |
if err != nil { | |
fmt.Printf("steam: could not ping %v: %v", addr, err) | |
continue | |
} | |
// we're good so far. lets get the server info | |
info, err := server.Info() | |
if err != nil { | |
fmt.Printf("steam: could not get server info from %v: %v", addr, err) | |
continue | |
} | |
// still good, phew! lets get the player info | |
playersInfo, err := server.PlayersInfo() | |
if err != nil { | |
fmt.Printf("steam: could not get players info from %v: %v", addr, err) | |
continue | |
} | |
// we've got all the info we need from the server. lets print out some stuff. | |
fmt.Printf(" Server | %v\n", info.Name) | |
fmt.Printf(" Name | %s\n", info.Game) | |
fmt.Printf(" Ping | %s\n", ping) | |
fmt.Printf(" Type | %s\n", info.ServerType) | |
fmt.Printf("Version | %s\n", info.Version) | |
fmt.Printf("Players | %d/%d\n", info.Players, info.MaxPlayers) | |
fmt.Printf(" Map | %s\n", info.Map) | |
fmt.Printf(" IP | %s\n", addr) | |
// if there's at least 1x player, lets print out their info | |
if len(playersInfo.Players) > 0 { | |
fmt.Printf("——–|—————————————–\n") | |
fmt.Printf(" Score | Time On | Player Name\n") | |
fmt.Printf("——–|—————————————–\n") | |
var temp SteamPlayers | |
temp.Players = playersInfo.Players | |
sort.Sort(sort.Reverse(temp)) | |
for _, player := range temp.Players { | |
fmt.Printf("%7d | %7d | %s\n", player.Score, int(player.Duration / 60), player.Name) | |
} | |
fmt.Printf("\n") | |
} | |
} | |
} | |
func must(err error) { | |
if err != nil { | |
panic(err) | |
} | |
} |
Problem: I want to get my Google Analytics data into my own “backend system”
Solution: an example of how to download google analytics data without any realtime user intervention required (no Auth windows etc), using a Google Cloud service account and Go.
there are a number of pre-requisite steps that must be completed before we can get Go-ing. lets get the non-Go steps done first.
click “Generate Certificate”. you’ll be prompted to download the private key. save this and keep it safe (you can only download it once, ever).
it will also tell you the private key’s password; make a note of this, as you’ll need it when converting the .p12 key you just downloaded into a .pem key (which is what Google’s OAuth requires for service accounts) later.
8961649gibberishkjxcjhbsdv@developer.gserviceaccount.com
), you should download the JSON file (this is your application’s “client secrets” file). you will set the variable gaServiceAcctSecretsFile to the location of the file later.$ openssl pkcs12 -in privatekeyfilename.p12 -nodes -nocerts > privatekeyfilename.pem
you’ll be asked for the Import password. enter the private key’s password you were given above. now you’ve got the PEM key! you will be setting the variable gaServiceAcctPEMKey to the location of this file later.
login there, navigate to the specific account/property/profile, and you will see the TableID in the “ids” box. it will look something like:
ga:12345678
make a note of it, you setting the variable gaTableID to this value later.
w00t! that’s all of the pre-requisite steps done & dusted. let’s look at some Go code (finally).
in order to use the Go Google API, it must be installed. to install all of the Go Google API’s (Drive, BigQuery, Calendar, etc), you can just do:
$ go get golang.org/x/oauth2
$ go get google.golang.org/api/analytics/v3
once you’ve ensured that you’ve got the above packages safely import-able in your $GOPATH, you’re good to go.
no more pre-requisite steps. let’s talk code.
head on over to htps://github.com/rickt/analyticsdumper. full instructions and example run are there! but here’s the sauce since that’s why we’re all here 🙂
UPDATED source:
package main | |
import ( | |
"fmt" | |
"golang.org/x/oauth2" | |
"golang.org/x/oauth2/jwt" | |
"google.golang.org/api/analytics/v3" | |
"io/ioutil" | |
"log" | |
"time" | |
) | |
// constants | |
const ( | |
datelayout string = "2006-01-02" // date format that Core Reporting API requires | |
) | |
// globals that you DON'T need to change | |
var ( | |
enddate string = time.Now().Format(datelayout) // set end query date to today | |
startdate string = time.Now().Add(time.Hour * 24 * –1).Format(datelayout) // set start query date to yesterday | |
metric string = "ga:pageviews" // GA metric that we want | |
tokenurl string = "https://accounts.google.com/o/oauth2/token" // (json:"token_uri") Google oauth2 Token URL | |
) | |
// globals that you DO need to change | |
// populate these with values from the JSON secretsfile obtained from the Google Cloud Console specific to your app) | |
// example secretsfile JSON: | |
// { | |
// "web": { | |
// "auth_uri": "https://accounts.google.com/o/oauth2/auth", | |
// "token_uri": "https://accounts.google.com/o/oauth2/token", | |
// "client_email": "blahblahblahblah@developer.gserviceaccount.com", | |
// "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/blahblahblahblah@developer.gserviceaccount.com", | |
// "client_id": "blahblahblahblah.apps.googleusercontent.com", | |
// "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs" | |
// } | |
// } | |
var ( | |
// CHANGE THESE!!! | |
gaServiceAcctEmail string = "blahblahblahblah@developer.gserviceaccount.com" // (json:"client_email") email address of registered application | |
gaServiceAcctPEMKey string = "./analyticsdumper.pem" // full path to private key file (PEM format) of your application from Google Cloud Console | |
gaTableID string = "ga:NNNNNNNN" // namespaced profile (table) ID of your analytics account/property/profile | |
) | |
// func: main() | |
// the main function. | |
func main() { | |
// load up the registered applications private key | |
pk, err := ioutil.ReadFile(gaServiceAcctPEMKey) | |
if err != nil { | |
log.Fatal("Error reading GA Service Account PEM key -", err) | |
} | |
// create a jwt.Config that we will subsequently use for our authenticated client/transport | |
// relevant docs for all the oauth2 & json web token stuff at https://godoc.org/golang.org/x/oauth2 & https://godoc.org/golang.org/x/oauth2/jwt | |
jwtc := jwt.Config{ | |
Email: gaServiceAcctEmail, | |
PrivateKey: pk, | |
Scopes: []string{analytics.AnalyticsReadonlyScope}, | |
TokenURL: tokenurl, | |
} | |
// create our authenticated http client using the jwt.Config we just created | |
clt := jwtc.Client(oauth2.NoContext) | |
// create a new analytics service by passing in the authenticated http client | |
as, err := analytics.New(clt) | |
if err != nil { | |
log.Fatal("Error creating Analytics Service at analytics.New() -", err) | |
} | |
// create a new analytics data service by passing in the analytics service we just created | |
// relevant docs for all the analytics stuff at https://godoc.org/google.golang.org/api/analytics/v3 | |
ads := analytics.NewDataGaService(as) | |
// w00t! now we're talking to the core reporting API. the hard stuff is over, lets setup a simple query. | |
// setup the query, call the Analytics API via our analytics data service's Get func with the table ID, dates & metric variables | |
gasetup := ads.Get(gaTableID, startdate, enddate, metric) | |
// send the query to the API, get a big fat gaData back. | |
gadata, err := gasetup.Do() | |
if err != nil { | |
log.Fatal("API error at gasetup.Do() -", err) | |
} | |
// print out some nice things | |
fmt.Printf("%s pageviews for %s (%s) from %s to %s.\n", gadata.Rows[0], gadata.ProfileInfo.ProfileName, gadata.ProfileInfo.WebPropertyId, startdate, enddate) | |
return | |
} |