mc sql
Syntax
The mc sql command provides an S3 Select interface for performing sql queries on objects in the specified MinIO deployment.
See Selecting content from objects for more information on S3 Select behavior and limitations.
The following command queries all objects in the mydata bucket on the myminio MinIO deployment:
mc sql --recursive --query "select * from S3Object" myminio/mydata
The command has the following syntax:
mc [GLOBALFLAGS] mc sql \
--query "string" \
[--csv-input "string"] \
[--compression "string"] \
[--csv-output "string"] \
[--csv-output-header "string"] \
[--enc-c "string"] \
[--json-input "string"] \
[--json-output "string"] \
[--recursive] \
ALIAS
- Brackets
[]indicate optional parameters. - Parameters sharing a line are mutually dependent.
- Parameters separated using the pipe
|operator are mutually exclusive.
Copy the example to a text editor and modify as-needed before running the command in the terminal/shell.
Parameters
ALIAS
mc-cmd
Required
The full path to the bucket or object to run the SQL query against. Specify the alias of a configured S3 service as the prefix to the ALIAS path. For example:
mc sql [FLAGS] play/mybucket
--query, e
mc-cmd
Required
The SQL statement to execute on the specified ALIAS directory or object. Wrap the entire SQL query in double quotes ".
Defaults to "select * from S3Object".
--csv-input
mc-cmd
Optional
The data format for .csv input objects. Specify a string of comma-seperated key=value,... pairs. See CSV Formatting Fields for more information on valid keys.
--compression
mc-cmd
Optional
The compression type of the input object. Specify one of the following supported values:
GZIPBZIP2NONE(default)
Compression schemes supported by MinIO backend only:
--csv-output
mc-cmd
Optional
The data format for .csv output. Specify a string of comma-seperated key=value,... pairs. See CSV Formatting Fields for more information on valid keys.
See the S3 API CSVOutput for more information.
--csv-output-header
mc-cmd
Optional
The header row of the .csv output file. Specify a string of comma-separated fields as field1,field2,....
Omit to output a .csv with no header row.
--enc-c
mc-cmd
Optional
Encrypt or decrypt objects using server-side SSE-C encryption with client-managed keys.
The parameter accepts a key-value pair formatted as KEY=VALUE
| The full path to the object as You can specify only the top-level path to use a single encryption key for all operations in that path. |
| Specify either a 32-byte RawBase64-encoded key or a 64-byte hex-encoded key for use with SSE-C encryption. Raw Base64 encoding rejects |
KEY- the full path to the object asalias/bucket/path/object.VALUE- the 32-byte RAW Base64-encoded data key to use for encrypting object(s).
For example:
# RawBase64-Encoded string "mybucket32byteencryptionkeyssec"
--enc-c "myminio/mybucket/prefix/object.obj=bXlidWNrZXQzMmJ5dGVlbmNyeXB0aW9ua2V5c3NlYwo"
You can specify multiple encryption keys by repeating the parameter.
Specify the path to a prefix to apply encryption to all matching objects at that path:
--enc-c "myminio/mybucket/prefix/=bXlidWNrZXQzMmJ5dGVlbmNyeXB0aW9ua2V5c3NlYwo"
Note
MinIO strongly recommends against using SSE-C encryption in production workloads. Use SSE-KMS via the --enc-kms or SSE-S3 via --enc-s3 parameters instead.
--json-input
mc-cmd
Optional
The data format for .json or .ndjson input objects. Specify the type of the JSON contents as type=<VALUE>. The value can be either:
See the S3 API JSONInput for more information.
--json-output
mc-cmd
Optional
The data format for the .json output. Supports the rd=value key, where rd is the RecordDelimiter for the JSON document.
Omit to use the default newline character \n.
See the S3 API JSONOutput for more information.
--recursive, r
mc-cmd
Optional
Recursively searches the specified ALIAS directory using the --query SQL statement.
Global Flags
This command supports any of the global flags.
Examples
Select all Columns in all Objects in a Bucket
Use mc sql with the --recursive and --query options to apply the query to all objects in a bucket:
mc sql --recursive --query "select * from S3Object" ALIAS/PATH
- Replace
ALIASwith the alias of the MinIO deployment. - Replace
PATHwith the path to the bucket on the MinIO deployment.
Run an Aggregation Query on an Object
Use mc sql with the --query option to query an object on an MinIO deployment:
mc sql --query "select count(s.power) from S3Object" ALIAS/PATH
- Replace
ALIASwith the alias of the MinIO deployment. - Replace
PATHwith the path to the object on the MinIO deployment.
Behavior
Input Formats
mc sql supports the following input formats:
| Type | content-type Value |
|---|---|
.csv | text/csv |
.json | application/json |
.parquet | none |
For .csv file types, use mc sql --csv-input to specify the CSV data format. See CSV Formatting Fields for more information on CSV formatting fields.
For .json file types, use mc sql --json-input to specify the JSON data format.
For .parquet file types, mc sql automatically interprets the data format.
mc sql determines the type by the file extension of the target object. For example, an object named data.json is interpreted as a JSON file.
You can query data of a supported type but a different extension if the object has the appropriate content-type. For more information, see mc cp --attr.
CSV Formatting Fields
The following table lists valid key-value pairs for use with mc sql --csv-input and mc sql --csv-output. Certain key pairs are only valid for --csv-input. See the documentation for S3 API CSVInput for more information on S3 CSV formatting.
Key |
| Description |
|---|---|---|
| The character that seperates each record (row) in the input Corresponds to | |
| The character that seperates each field in a record. Defaults to Corresponds to | |
| The character used for escaping when the Corresponds to | |
| The character used for escaping a quotation mark Corresponds to | |
| Yes | The content of the first line in the Specify one of the following supported values:
For For Corresponds to |
| Yes | The character used to indicate a record should be ignored. The character must appear at the beginning of the record. Corresponds to |
| Yes | Specify Defaults to Corresponds to |
S3 Compatibility
The mc commandline tool is built for compatibility with the AWS S3 API and is tested with MinIO and AWS S3 for expected functionality and behavior.
MinIO provides no guarantees for other S3-compatible services, as their S3 API implementation is unknown and therefore unsupported. While mc commands may work as documented, any such usage is at your own risk.
Portions of this page are adapted from the MinIO Object Storage Documentation, © 2020–Present MinIO, Inc., licensed under CC BY 4.0, converted and maintained by the Silo project. Attribution