jq (programming language)

jq was created by Stephen Dolan, and released in October 2012.{{sfn|Janssens|2014}}{{Cite web |title=jq |website=jq |url=https://github.com/jqlang/jq/blob/master/NEWS |access-date=January 6, 2023}}

It was described as being "like sed for JSON data".{{Cite web|title=like sed|url=http://jqlang.github.io/jq|archive-url=https://web.archive.org/web/20130414054332/http://stedolan.github.io/jq |archive-date=2013-04-14}} Support for regular expressions was added in jq version 1.5.

A "wrapper" program for jq named [https://github.com/kislyuk/yq yq] adds support for YAML, XML and TOML. It was first released in 2017.{{Cite web |title=Release v2.0.0 · kislyuk/yq |website=GitHub |url=https://github.com/kislyuk/yq/releases/tag/v2.0.0}}

The Go implementation, gojq, was initially released in 2019.{{Cite web|title=Release v0.0.1 · itchyny/gojq |website=GitHub |url=https://github.com/itchyny/gojq/releases/tag/v0.0.1}} gojq notably extends jq to include support for YAML.

The Rust implementation, jaq, has as its project goals a faster and more correct implementation of jq, while preserving compatibility with jq in most cases. Explicitly excluded from the project goals as of March 2024 are certain advanced features of jq such as modules, SQL-style operators, and a streaming parser for very large JSON documents.{{Cite web|title=01mf02/jaq: A jq clone focussed on correctness, speed, and simplicity |website=GitHub |url=https://github.com/01mf02/jaq |access-date=March 6, 2024}}

The jq implementation, jqjq, was initially released in 2022. jqjq notably can run itself, has a REPL and supports eval.

jq is typically used at the command line and can be used with other command-line utilities, such as curl. Here is an example showing how the output of a {{code|curl}} command can be piped to a jq filter to determine the category names associated with this Wikipedia page:

$ curl -s 'https://en.wikipedia.org/w/api.php?action=parse&page=jq_(programming_language)&format=json' | jq '.parse.categories[]."*"'

The output produced by this pipeline consists of a stream of JSON strings, the first few of which are:

"Articles_with_short_description"

"Short_description_matches_Wikidata"

"Dynamically_typed_programming_languages"

"Functional_languages"

"Programming_languages"

"Programming_languages_created_in_2012"

"Query_languages"

"2012_software"

The {{code|curl}} command above uses the MediaWiki API for this page to produce a JSON response.

The pipe {{code|{{!}}}} allows the output of {{code|curl}} to be accessed by jq, a standard Unix shell mechanism.{{Cite web |title=Tutorial |website=jq |url=https://jqlang.github.io/jq/tutorial/ |access-date=January 6, 2023}}

The jq filter shown is an abbreviation for the jq pipeline:

.["parse"] | .["categories"] | .[] | .["*"]

This corresponds to the nested JSON structure produced by the call to {{code|curl}}. Notice that the jq pipeline is constructed in the same manner using the {{code|{{!}}}} character as the Unix-style pipeline.

Both the C and the Go implementations provide libraries so that jq functions can be embedded in other applications and programming environments.

For example, gojq has been integrated with SQLite so that a {{code|jq}} function is available in SQL statements.{{Cite web |title=sqlite_jq |website=GitHub |url=https://github.com/mgdm/sqlite-jq}} This function is marked as

[https://www.sqlite.org/expridx.html "deterministic"] and

can therefore be used in "CREATE INDEX" commands.{{Cite web |title=FAQ |website=GitHub |url=https://github.com/jqlang/jq/wiki/FAQ#sqlite-bindings}}

jq by default acts as a "stream editor" for JSON inputs, much

like the sed utility can be thought of as a "stream editor" for lines of text.

However jq has several other modes of operation:

1. it can treat its input from one or more sources as lines of text;
2. it can gather a stream of inputs from a specified source into a JSON array;
3. it can parse its JSON inputs using a so-called "streaming parser" that produces a stream of [path, value] arrays for all "leaf" paths.

The streaming parser is very useful when one of more of the JSON inputs is too large to fit in memory, since its memory needs are usually quite small. For example, for an arbitrarily large array of JSON objects, the peak memory need is little more than needed to handle the largest top-level object.

These modes of operation can, within certain limitations, be combined.

Every JSON value is also a value in jq, which accordingly has the data types shown in the table below.{{Cite web |title=Manual |website=jq |url=https://jqlang.github.io/jq/manual/ |access-date=January 6, 2023}} The gojq and jaq implementations distinguish between integers and non-integer numbers. The gojq implementation supports unbounded-precision integer arithmetic, as did the original implementation of jq in Haskell.

[https://www.json.org/json-en.html {{code|null}}] is a value, just like any other JSON scalar; it is not a pointer or a "null-pointer".

{{code|nan}} (corresponding to NaN) and {{code|infinite}} (see IEEE 754) are the only two jq scalars that are not also JSON values.

Special syntactic forms exist for function creation, conditionals, stream reduction, and the module system.

Here is an example of defining a named, parameterized filter for formatting an integer in any base from 2 to 36 inclusive. The implementation illustrates tacit (or point-free) programming:

1. Use gojq for infinite precision integer arithmetic

def tobase($b):

def digit: "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"[.:.+1];

def mod: . % $b;

def div: ((. - mod) / $b);

def digits: recurse( select(. >= $b) | div) | mod ;

select(2 <= $b and $b <= 36)

| [digits | digit] | reverse | add;

The next example demonstrates the use of generators in the classic "SEND MORE MONEY" verbal arithmetic game:

def send_more_money:

def choose(m;n;used): ([range(m;n+1)] - used)[];

def num(a;b;c;d): 1000*a + 100*b + 10*c + d;

def num(a;b;c;d;e): 10*num(a;b;c;d) + e;

first(

1 as $m

| 0 as $o

| choose(8;9;[]) as $s

| choose(2;9;[$s]) as $e

| choose(2;9;[$s,$e]) as $n

| choose(2;9;[$s,$e,$n]) as $d

| choose(2;9;[$s,$e,$n,$d]) as $r

| choose(2;9;[$s,$e,$n,$d,$r]) as $y

| select(num($s;$e;$n;$d) + num($m;$o;$r;$e) ==

num($m;$o;$n;$e;$y))

| [$s,$e,$n,$d,$m,$o,$r,$e,$m,$o,$n,$e,$y] );

There is a very close relationship between jq and the parsing expression grammar (PEG) formalism.

{{Cite web |title=PEG |website=PEG |url=https://github.com/jqlang/jq/wiki/Parsing-Expression-Grammars}}

The relationship stems from the equivalence of the seven basic PEG operations and the jq constructs shown in the following table.

gojq is a pure Go implementation. There is also a Rust implementation of a dialect of jq named jaq for which a denotational semantics has been specified.

{{cite arXiv

|last1=Färber |first1=Michael |date=2023

|title=Denotational Semantics and a fast interpreter for jq

|class=cs.LO

|eprint=2302.10576

}}

{{Reflist|group=note}}

{{Refbegin}}

• {{Cite book |last1=Janssens |first1=Jeroen |date=2021 |url=https://books.google.com/books?id=-hg-EAAAQBAJ |title=Data Science at the Command Line |publisher=O'Reilly Media |isbn=9781492087885 |language=en}}
• {{Cite book |last1=Janssens |first1=Jeroen |date=2014 |url=https://books.google.com/books?id=EsWeBAAAQBAJ |title=Data Science at the Command Line: Facing the Future with Time-Tested Tools |publisher=O'Reilly Media |isbn=9781491947807 |language=en}}
• {{Cite book |last1=Marrs |first1=Tom |date=2017 |url=https://books.google.com/books?id=e6woDwAAQBAJ |title=JSON at Work: Practical Data Integration for the Web |publisher=O'Reilly Media |isbn=9781491982419 |language=en}}

{{Refend}}

{{Reflist}}

• {{Official website|jqlang.org}}
• {{GitHub|itchyny/gojq gojq}} – Pure Go implementation
• {{GitHub|01mf02/jaq jaq}} – Rust implementation
• {{GitHub|wader/jqjq jqjq}} – jq implementation
• {{GitHub|jqlang/jq/wiki/FAQ}} – jq FAQ
• {{GitHub|fiatjaf/awesome-jq}} – Awesome jq - curated list of jq-related resources
• [http://rosettacode.org/wiki/Category:Jq The jq Programming Language page] on the Rosetta Code comparative programming tasks project site

{{DEFAULTSORT:jq (Programming Language)}}

Category:Dynamically typed programming languages

Category:Functional languages

Category:Programming languages

Category:Programming languages created in 2012

Category:Query languages

Category:2012 software

Category:Articles with example code