Introduction

gq is a command line filter (à la sed/awk) used for rendering Go templates, along with a few other useful data transformation features. It is heavily inspired by Kubectl and Helm.

Features

Go template rendering (Sprig functions included)
JSONPath querying (using Kubectl syntax)
Data format transformation
Line-based processing and whole-input processing

Quick Start

Fetch a GitHub repo's tags and parse the response:

curl --silent https://api.github.com/repos/torvalds/linux/git/refs/tags \
| gq -r 'printf "%s -> %s\n" (.ref | replace "refs/tags/" "") .object.sha'

Usage

gq supports a few core features, which are explained in-depth in the following chapters.

By default (ie, given no arguments or flags), gq will read from standard input until it encounters an end-of-file (EOF), expecting the data to be in JSON format. It then injects this input data into a trivial Go template (noted below) and renders the result to standard output. Of course, this behavior can be modified using the following options.

Positional Arguments

`[template string]`

Either a Go template, or a JSONPath query (using kubectl syntax).

Default:

Go template: {{ . }}
JSONPath: { . }

Flags

`-f, --file`

File to read input from.

Default: - (stdin)

`-i, --input`

Input format.

Options:

json (default)
yaml
toml
hcl
- Note: At this time, only HCL 1 (Terraform <= 0.11) is supported. HCL 2 (Terraform >= 0.12) support is planned.

`-o, --output`

Output format.

Options:

go-template (default)
jsonpath
json
yaml
toml

`-l, --lines`

Apply the operation to each line, rather than the whole input as one.

Default: false

`-s, --simple`

Automatically wraps the template string with the necessary delimiters if they do not exist. For Go templates this is {{ }}, while for JSONPath it is { }.

Default: true

`-v, --version`

Print program version information and exit. This flag overrides all others.

Default: false

Examples

The next few chapters explore some examples of how gq can be used in the wild to increase productivity!

Log Processing

HashiCorp Vault Audit Log

HashiCorp Vault can write JSON-formatted audit logs.

gq can be used to process a stream of these logs to CSV (for example). All extra whitespace in error messages is compressed to a single space and trimmed.

curl -sL https://github.com/hashicorp/vault-guides/raw/master/monitoring-troubleshooting/vault-audit.log \
| gq -l 'list .time .type .request.path .request.operation (regexReplaceAll "\\s+" (default "" .error | trim) " ") | join ","'

Which will output the following (trimmed to the last 5 lines for brevity):

2020-04-30T19:12:58.5648483Z,response,sys/mounts,read,1 error occurred: * permission denied
2020-04-30T19:32:00.7744629Z,request,auth/userpass/login/lab-user-4,update,
2020-04-30T19:32:00.9207237Z,response,auth/userpass/login/lab-user-4,update,
2020-04-30T19:35:23.1771431Z,request,auth/userpass/login/lab-user-5,update,
2020-04-30T19:35:23.2895529Z,response,auth/userpass/login/lab-user-5,update,

gq Documentation