mirror of https://git.sr.ht/~hrbrmstr/awsathena
boB Rudis
5 years ago
commit
1d92afcf6f
35 changed files with 1263 additions and 0 deletions
@ -0,0 +1,13 @@ |
|||
^LICENSE\.md$ |
|||
^.*\.Rproj$ |
|||
^\.Rproj\.user$ |
|||
^\.travis\.yml$ |
|||
^README\.*Rmd$ |
|||
^README\.*html$ |
|||
^NOTES\.*Rmd$ |
|||
^NOTES\.*html$ |
|||
^\.codecov\.yml$ |
|||
^README_files$ |
|||
^docs$ |
|||
^CONDUCT\.md$ |
|||
^\.bash_profile$ |
@ -0,0 +1 @@ |
|||
comment: false |
@ -0,0 +1,8 @@ |
|||
.DS_Store |
|||
.Rproj.user |
|||
.Rhistory |
|||
.RData |
|||
.Rproj |
|||
src/*.o |
|||
src/*.so |
|||
src/*.dll |
@ -0,0 +1,8 @@ |
|||
language: r |
|||
cache: packages |
|||
r_packages: |
|||
- covr |
|||
r_github_packages: |
|||
- hrbrmstr/jerichojars |
|||
after_success: |
|||
- Rscript -e 'library("covr");codecov()' |
@ -0,0 +1,30 @@ |
|||
Package: awsathena |
|||
Type: Package |
|||
Title: rJava Interface to AWS Athena SDK |
|||
Version: 0.1.0 |
|||
Date: 2019-02-22 |
|||
Authors@R: c( |
|||
person("Bob", "Rudis", role = c("aut", "cre"), email = "bob@rud.is") |
|||
) |
|||
Maintainer: Bob Rudis <bob@rud.is> |
|||
SystemRequirements: Java |
|||
Description: Provides R wrapper methods to core 'aws-java-sdk-athena' Java library methods |
|||
to interrogate and orchestrate Amazon Athena queries. |
|||
URL: https://github.com/hrbrmstr/awsathena |
|||
BugReports: https://github.com/hrbrmstr/awsathena/issues |
|||
License: Apache License 2.0 | file LICENSE |
|||
Encoding: UTF-8 |
|||
Suggests: |
|||
testthat, |
|||
covr, |
|||
dbplyr |
|||
Depends: |
|||
R (>= 3.2.0), |
|||
rJava, |
|||
awsathenajars |
|||
Imports: |
|||
uuid, |
|||
utils |
|||
RoxygenNote: 6.1.1 |
|||
Remotes: |
|||
hrbrmstr/awsathenajars |
@ -0,0 +1,201 @@ |
|||
Apache License |
|||
Version 2.0, January 2004 |
|||
http://www.apache.org/licenses/ |
|||
|
|||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION |
|||
|
|||
1. Definitions. |
|||
|
|||
"License" shall mean the terms and conditions for use, reproduction, |
|||
and distribution as defined by Sections 1 through 9 of this document. |
|||
|
|||
"Licensor" shall mean the copyright owner or entity authorized by |
|||
the copyright owner that is granting the License. |
|||
|
|||
"Legal Entity" shall mean the union of the acting entity and all |
|||
other entities that control, are controlled by, or are under common |
|||
control with that entity. For the purposes of this definition, |
|||
"control" means (i) the power, direct or indirect, to cause the |
|||
direction or management of such entity, whether by contract or |
|||
otherwise, or (ii) ownership of fifty percent (50%) or more of the |
|||
outstanding shares, or (iii) beneficial ownership of such entity. |
|||
|
|||
"You" (or "Your") shall mean an individual or Legal Entity |
|||
exercising permissions granted by this License. |
|||
|
|||
"Source" form shall mean the preferred form for making modifications, |
|||
including but not limited to software source code, documentation |
|||
source, and configuration files. |
|||
|
|||
"Object" form shall mean any form resulting from mechanical |
|||
transformation or translation of a Source form, including but |
|||
not limited to compiled object code, generated documentation, |
|||
and conversions to other media types. |
|||
|
|||
"Work" shall mean the work of authorship, whether in Source or |
|||
Object form, made available under the License, as indicated by a |
|||
copyright notice that is included in or attached to the work |
|||
(an example is provided in the Appendix below). |
|||
|
|||
"Derivative Works" shall mean any work, whether in Source or Object |
|||
form, that is based on (or derived from) the Work and for which the |
|||
editorial revisions, annotations, elaborations, or other modifications |
|||
represent, as a whole, an original work of authorship. For the purposes |
|||
of this License, Derivative Works shall not include works that remain |
|||
separable from, or merely link (or bind by name) to the interfaces of, |
|||
the Work and Derivative Works thereof. |
|||
|
|||
"Contribution" shall mean any work of authorship, including |
|||
the original version of the Work and any modifications or additions |
|||
to that Work or Derivative Works thereof, that is intentionally |
|||
submitted to Licensor for inclusion in the Work by the copyright owner |
|||
or by an individual or Legal Entity authorized to submit on behalf of |
|||
the copyright owner. For the purposes of this definition, "submitted" |
|||
means any form of electronic, verbal, or written communication sent |
|||
to the Licensor or its representatives, including but not limited to |
|||
communication on electronic mailing lists, source code control systems, |
|||
and issue tracking systems that are managed by, or on behalf of, the |
|||
Licensor for the purpose of discussing and improving the Work, but |
|||
excluding communication that is conspicuously marked or otherwise |
|||
designated in writing by the copyright owner as "Not a Contribution." |
|||
|
|||
"Contributor" shall mean Licensor and any individual or Legal Entity |
|||
on behalf of whom a Contribution has been received by Licensor and |
|||
subsequently incorporated within the Work. |
|||
|
|||
2. Grant of Copyright License. Subject to the terms and conditions of |
|||
this License, each Contributor hereby grants to You a perpetual, |
|||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable |
|||
copyright license to reproduce, prepare Derivative Works of, |
|||
publicly display, publicly perform, sublicense, and distribute the |
|||
Work and such Derivative Works in Source or Object form. |
|||
|
|||
3. Grant of Patent License. Subject to the terms and conditions of |
|||
this License, each Contributor hereby grants to You a perpetual, |
|||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable |
|||
(except as stated in this section) patent license to make, have made, |
|||
use, offer to sell, sell, import, and otherwise transfer the Work, |
|||
where such license applies only to those patent claims licensable |
|||
by such Contributor that are necessarily infringed by their |
|||
Contribution(s) alone or by combination of their Contribution(s) |
|||
with the Work to which such Contribution(s) was submitted. If You |
|||
institute patent litigation against any entity (including a |
|||
cross-claim or counterclaim in a lawsuit) alleging that the Work |
|||
or a Contribution incorporated within the Work constitutes direct |
|||
or contributory patent infringement, then any patent licenses |
|||
granted to You under this License for that Work shall terminate |
|||
as of the date such litigation is filed. |
|||
|
|||
4. Redistribution. You may reproduce and distribute copies of the |
|||
Work or Derivative Works thereof in any medium, with or without |
|||
modifications, and in Source or Object form, provided that You |
|||
meet the following conditions: |
|||
|
|||
(a) You must give any other recipients of the Work or |
|||
Derivative Works a copy of this License; and |
|||
|
|||
(b) You must cause any modified files to carry prominent notices |
|||
stating that You changed the files; and |
|||
|
|||
(c) You must retain, in the Source form of any Derivative Works |
|||
that You distribute, all copyright, patent, trademark, and |
|||
attribution notices from the Source form of the Work, |
|||
excluding those notices that do not pertain to any part of |
|||
the Derivative Works; and |
|||
|
|||
(d) If the Work includes a "NOTICE" text file as part of its |
|||
distribution, then any Derivative Works that You distribute must |
|||
include a readable copy of the attribution notices contained |
|||
within such NOTICE file, excluding those notices that do not |
|||
pertain to any part of the Derivative Works, in at least one |
|||
of the following places: within a NOTICE text file distributed |
|||
as part of the Derivative Works; within the Source form or |
|||
documentation, if provided along with the Derivative Works; or, |
|||
within a display generated by the Derivative Works, if and |
|||
wherever such third-party notices normally appear. The contents |
|||
of the NOTICE file are for informational purposes only and |
|||
do not modify the License. You may add Your own attribution |
|||
notices within Derivative Works that You distribute, alongside |
|||
or as an addendum to the NOTICE text from the Work, provided |
|||
that such additional attribution notices cannot be construed |
|||
as modifying the License. |
|||
|
|||
You may add Your own copyright statement to Your modifications and |
|||
may provide additional or different license terms and conditions |
|||
for use, reproduction, or distribution of Your modifications, or |
|||
for any such Derivative Works as a whole, provided Your use, |
|||
reproduction, and distribution of the Work otherwise complies with |
|||
the conditions stated in this License. |
|||
|
|||
5. Submission of Contributions. Unless You explicitly state otherwise, |
|||
any Contribution intentionally submitted for inclusion in the Work |
|||
by You to the Licensor shall be under the terms and conditions of |
|||
this License, without any additional terms or conditions. |
|||
Notwithstanding the above, nothing herein shall supersede or modify |
|||
the terms of any separate license agreement you may have executed |
|||
with Licensor regarding such Contributions. |
|||
|
|||
6. Trademarks. This License does not grant permission to use the trade |
|||
names, trademarks, service marks, or product names of the Licensor, |
|||
except as required for reasonable and customary use in describing the |
|||
origin of the Work and reproducing the content of the NOTICE file. |
|||
|
|||
7. Disclaimer of Warranty. Unless required by applicable law or |
|||
agreed to in writing, Licensor provides the Work (and each |
|||
Contributor provides its Contributions) on an "AS IS" BASIS, |
|||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
|||
implied, including, without limitation, any warranties or conditions |
|||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A |
|||
PARTICULAR PURPOSE. You are solely responsible for determining the |
|||
appropriateness of using or redistributing the Work and assume any |
|||
risks associated with Your exercise of permissions under this License. |
|||
|
|||
8. Limitation of Liability. In no event and under no legal theory, |
|||
whether in tort (including negligence), contract, or otherwise, |
|||
unless required by applicable law (such as deliberate and grossly |
|||
negligent acts) or agreed to in writing, shall any Contributor be |
|||
liable to You for damages, including any direct, indirect, special, |
|||
incidental, or consequential damages of any character arising as a |
|||
result of this License or out of the use or inability to use the |
|||
Work (including but not limited to damages for loss of goodwill, |
|||
work stoppage, computer failure or malfunction, or any and all |
|||
other commercial damages or losses), even if such Contributor |
|||
has been advised of the possibility of such damages. |
|||
|
|||
9. Accepting Warranty or Additional Liability. While redistributing |
|||
the Work or Derivative Works thereof, You may choose to offer, |
|||
and charge a fee for, acceptance of support, warranty, indemnity, |
|||
or other liability obligations and/or rights consistent with this |
|||
License. However, in accepting such obligations, You may act only |
|||
on Your own behalf and on Your sole responsibility, not on behalf |
|||
of any other Contributor, and only if You agree to indemnify, |
|||
defend, and hold each Contributor harmless for any liability |
|||
incurred by, or claims asserted against, such Contributor by reason |
|||
of your accepting any such warranty or additional liability. |
|||
|
|||
END OF TERMS AND CONDITIONS |
|||
|
|||
APPENDIX: How to apply the Apache License to your work. |
|||
|
|||
To apply the Apache License to your work, attach the following |
|||
boilerplate notice, with the fields enclosed by brackets "[]" |
|||
replaced with your own identifying information. (Don't include |
|||
the brackets!) The text should be enclosed in the appropriate |
|||
comment syntax for the file format. We also recommend that a |
|||
file or class name and description of purpose be included on the |
|||
same "printed page" as the copyright notice for easier |
|||
identification within third-party archives. |
|||
|
|||
Copyright [yyyy] [name of copyright owner] |
|||
|
|||
Licensed under the Apache License, Version 2.0 (the "License"); |
|||
you may not use this file except in compliance with the License. |
|||
You may obtain a copy of the License at |
|||
|
|||
http://www.apache.org/licenses/LICENSE-2.0 |
|||
|
|||
Unless required by applicable law or agreed to in writing, software |
|||
distributed under the License is distributed on an "AS IS" BASIS, |
|||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
|||
See the License for the specific language governing permissions and |
|||
limitations under the License. |
@ -0,0 +1,14 @@ |
|||
# Generated by roxygen2: do not edit by hand |
|||
|
|||
export(collect_async) |
|||
export(download_query_execution_results) |
|||
export(get_query_execution) |
|||
export(list_query_executions) |
|||
export(s3_download_file) |
|||
export(start_query_execution) |
|||
export(stop_query_execution) |
|||
import(awsathenajars) |
|||
import(rJava) |
|||
importFrom(utils,setTxtProgressBar) |
|||
importFrom(utils,txtProgressBar) |
|||
importFrom(uuid,UUIDgenerate) |
@ -0,0 +1,5 @@ |
|||
0.2.0 |
|||
* Split into two packages as per CRAN rJava-package suggested practice. |
|||
|
|||
0.1.0 |
|||
* Initial release |
@ -0,0 +1,49 @@ |
|||
aws_athena_client <- function(region = "us-east-1", profile = NULL, properties_file = NULL) { |
|||
|
|||
regions <- J("com.amazonaws.regions.Regions") |
|||
|
|||
default_creds <- J(class = "com.amazonaws.auth.DefaultAWSCredentialsProviderChain") |
|||
|
|||
client_builder <- .jnew("com.amazonaws.services.athena.AmazonAthenaClientBuilder") |
|||
|
|||
builder <- client_builder$standard() |
|||
builder <- builder$withRegion(.jawsregion(region)) |
|||
|
|||
if (!is.null(profile)) { |
|||
builder <- builder$withCredentials( |
|||
.jnew("com.amazonaws.auth.profile.ProfileCredentialsProvider", profile) |
|||
) |
|||
} else if (!is.null(properties_file)) { |
|||
builder <- builder$withCredentials(properties_file) |
|||
} else { |
|||
builder <- builder$withCredentials(default_creds$getInstance()) |
|||
} |
|||
|
|||
client <- builder$build() |
|||
|
|||
} |
|||
|
|||
aws_s3_client <- function(region = "us-east-1", profile = NULL, properties_file = NULL) { |
|||
|
|||
regions <- J("com.amazonaws.regions.Regions") |
|||
|
|||
default_creds <- J(class = "com.amazonaws.auth.DefaultAWSCredentialsProviderChain") |
|||
|
|||
client_builder <- .jnew("com.amazonaws.services.s3.AmazonS3ClientBuilder") |
|||
|
|||
builder <- client_builder$standard() |
|||
builder <- builder$withRegion(.jawsregion(region)) |
|||
|
|||
if (!is.null(profile)) { |
|||
builder <- builder$withCredentials( |
|||
.jnew("com.amazonaws.auth.profile.ProfileCredentialsProvider", profile) |
|||
) |
|||
} else if (!is.null(properties_file)) { |
|||
builder <- builder$withCredentials(properties_file) |
|||
} else { |
|||
builder <- builder$withCredentials(default_creds$getInstance()) |
|||
} |
|||
|
|||
client <- builder$build() |
|||
|
|||
} |
@ -0,0 +1,14 @@ |
|||
#' rJava Client to AWS Athena SDK |
|||
#' |
|||
#' Provides R wrapper methods to core 'aws-java-sdk-athena' Java library methods |
|||
#' to interrogate and orchestrate Amazon Athena queries. |
|||
#' |
|||
#' @md |
|||
#' @name awsathena |
|||
#' @docType package |
|||
#' @keywords internal |
|||
#' @author Bob Rudis (bob@@rud.is) |
|||
#' @importFrom uuid UUIDgenerate |
|||
#' @importFrom utils txtProgressBar setTxtProgressBar |
|||
#' @import rJava awsathenajars |
|||
NULL |
@ -0,0 +1,92 @@ |
|||
#' Collect Amazon Athena `dplyr` query results asynchronously |
|||
#' |
|||
#' Long running Athena queries and Athena queries with large result |
|||
#' sets can seriously stall a `dplyr` processing chain due to poorly |
|||
#' implemented ODBC and JDBC drivers. This function converts a `dplyr` |
|||
#' chain to a raw SQL query then submits it via [start_query_execution()] and |
|||
#' returns the query execution id. You can retrieve the result set either |
|||
#' via [download_query_execution_results()] or your favorite S3 downloading |
|||
#' method. |
|||
#' |
|||
#' @md |
|||
#' @param obj the `dplyr` query |
|||
#' @param database database within which the query executes. |
|||
#' @param output_location location in S3 where query results are stored. |
|||
#' @param client_request_token unique case-sensitive string used to ensure the |
|||
#' request to create the query is idempotent (executes only once). If another |
|||
#' `StartQueryExecution` request is received, the same response is returned |
|||
#' and another query is not created. If a parameter has changed, for example, |
|||
#' the `query` , an error is returned. **This is auto-generated-for-you**. |
|||
#' @param encryption_option indicates whether Amazon S3 server-side encryption |
|||
#' with Amazon S3-managed keys (`SSE-S3`), server-side encryption with |
|||
#' KMS-managed keys (`SSE-KMS`), or client-side encryption with KMS-managed |
|||
#' keys (`CSE-KMS`) is used. Default is `NULL` (no encryption) |
|||
#' @param kms_key For `SSE-KMS` and `CSE-KMS`, this is the KMS key ARN or ID. |
|||
#' Default is `NULL` (no encryption) |
|||
#' @param region AWS region string |
|||
#' @param profile if not using the default credentials chain or a dedicated |
|||
#' properties file then provide the named profile from `~/.aws/credentials` |
|||
#' you wish to use |
|||
#' @param properties_file if not using the default credentials provider chain or |
|||
#' a named profile then provide the path to an Athena credentials proeprty file. |
|||
#' @note `dbplyr` must be installed for this to work. It is not listed in |
|||
#' the `Imports` as it brings with it many dependencies that may not |
|||
#' be necessary for general use of this package. |
|||
#' @export |
|||
#' @examples \dontrun{ |
|||
#' library(odbc) |
|||
#' library(DBI) |
|||
#' |
|||
#' DBI::dbConnect( |
|||
#' odbc::odbc(), |
|||
#' driver = "/Library/simba/athenaodbc/lib/libathenaodbc_sbu.dylib", |
|||
#' Schema = "sampledb", |
|||
#' AwsRegion = "us-east-1", |
|||
#' AwsProfile = "personal", |
|||
#' AuthenticationType = "IAM Profile", |
|||
#' S3OutputLocation = "s3://aws-athena-query-results-redacted" |
|||
#' ) -> con |
|||
#' |
|||
#' elb_logs <- tbl(con, "elb_logs") |
|||
#' |
|||
#' mutate(elb_logs, tsday = substr(timestamp, 1, 10)) %>% |
|||
#' filter(tsday == "2014-09-29") %>% |
|||
#' select(requestip, requestprocessingtime) %>% |
|||
#' collect_async( |
|||
#' database = "sampledb", |
|||
#' output_location = "s3://aws-athena-query-results-redacted", |
|||
#' profile = "personal" |
|||
#' ) -> id |
|||
#' |
|||
#' get_query_execution(id, profile = "personal") |
|||
#' |
|||
#' # do this to get the data or use your favorite way to grab files from S3 |
|||
#' get_query_results(id, profile = "personal") |
|||
#' } |
|||
collect_async <- function(obj, |
|||
database, |
|||
output_location, |
|||
client_request_token = uuid::UUIDgenerate(), |
|||
encryption_option = NULL, |
|||
kms_key = NULL, |
|||
region = "us-east-1", |
|||
profile = NULL, |
|||
properties_file = NULL) { |
|||
|
|||
if (!requireNamespace("dbplyr", quietly = TRUE)) { |
|||
stop("dbplyr package required for this function", call. = FALSE) |
|||
} else { |
|||
start_query_execution( |
|||
query = as.character(dbplyr::sql_render(obj)), |
|||
database = database, |
|||
output_location = output_location, |
|||
client_request_token = client_request_token, |
|||
encryption_option = encryption_option, |
|||
kms_key = kms_key, |
|||
region = region, |
|||
profile = profile, |
|||
properties_file = properties_file |
|||
) |
|||
} |
|||
|
|||
} |
@ -0,0 +1,48 @@ |
|||
#' Use S3 to download the results of an Athena Query |
|||
#' |
|||
#' This method is much faster than the paginated interface |
|||
#' |
|||
#' @md |
|||
#' @param qxid query execution id |
|||
#' @param output_dir either a full or relative path ([path.expand()] will be run on |
|||
#' the value) to where you want the results to be stored. |
|||
#' @param progress show download progress? |
|||
#' @param region AWS region string |
|||
#' @param profile if not using the default credentials chain or a dedicated |
|||
#' properties file then provide the named profile from `~/.aws/credentials` |
|||
#' you wish to use |
|||
#' @param properties_file if not using the default credentials provider chain or |
|||
#' a named profile then provide the path to an Athena credentials proeprty file. |
|||
#' @export |
|||
download_query_execution_results <- function(qxid, |
|||
output_dir, |
|||
progress = FALSE, |
|||
region = "us-east-1", |
|||
profile = NULL, |
|||
properties_file = NULL) { |
|||
|
|||
if (missing(output_dir)) output_dir <- getwd() |
|||
|
|||
output_dir <- path.expand(output_dir) |
|||
|
|||
stopifnot(dir.exists(output_dir)) |
|||
|
|||
get_query_execution( |
|||
qxid, region = region, profile = profile, properties_file = properties_file |
|||
) -> qx_info |
|||
|
|||
s3_download_file( |
|||
bucket = gsub("^s3://", "", dirname(qx_info$output_location)), |
|||
key = basename(qx_info$output_location), |
|||
output_dir = output_dir, |
|||
progress = progress, |
|||
region = region, |
|||
profile = profile, |
|||
properties_file = properties_file |
|||
) -> res |
|||
|
|||
message("Results saved to ", res) |
|||
|
|||
invisible(res) |
|||
|
|||
} |
@ -0,0 +1,51 @@ |
|||
#' Get Query Execution |
|||
#' |
|||
#' @md |
|||
#' @param qxid query execution id |
|||
#' @param region AWS region string |
|||
#' @param profile if not using the default credentials chain or a dedicated |
|||
#' properties file then provide the named profile from `~/.aws/credentials` |
|||
#' you wish to use |
|||
#' @param properties_file if not using the default credentials provider chain or |
|||
#' a named profile then provide the path to an Athena credentials proeprty file. |
|||
#' @export |
|||
get_query_execution <- function(qxid, region = "us-east-1", profile = NULL, properties_file = NULL) { |
|||
|
|||
client <- aws_athena_client(region = region, profile = profile, properties_file = properties_file) |
|||
|
|||
qx_req <- .jnew("com.amazonaws.services.athena.model.GetQueryExecutionRequest") |
|||
qx_req <- qx_req$withQueryExecutionId(qxid) |
|||
|
|||
res <- client$getQueryExecution(qx_req) |
|||
|
|||
res <- res$getQueryExecution() |
|||
|
|||
status <- res$getStatus() |
|||
|
|||
submitted <- status$getSubmissionDateTime() |
|||
if (!is.jnull(submitted)) submitted <- submitted$toGMTString() |
|||
|
|||
completed <- status$getCompletionDateTime() |
|||
if (!is.jnull(completed)) completed <- completed$toGMTString() |
|||
|
|||
rcfg <- res$getResultConfiguration() |
|||
stats <- res$getStatistics() |
|||
|
|||
list( |
|||
workgroup = res$getWorkGroup(), |
|||
query = res$getQuery(), |
|||
context = res$getQueryExecutionContext()$getDatabase(), |
|||
statement_type = res$getStatementType(), |
|||
state = status$getState(), |
|||
started = submitted, |
|||
completed = completed, |
|||
output_location = rcfg$getOutputLocation(), |
|||
data_scanned = stats$getDataScannedInBytes(), |
|||
execution_time = stats$getEngineExecutionTimeInMillis() |
|||
) -> out |
|||
|
|||
client$shutdown() |
|||
|
|||
out |
|||
|
|||
} |
@ -0,0 +1,48 @@ |
|||
#' List Query Executions |
|||
#' |
|||
#' @md |
|||
#' @param region AWS region string |
|||
#' @param profile if not using the default credentials chain or a dedicated |
|||
#' properties file then provide the named profile from `~/.aws/credentials` |
|||
#' you wish to use |
|||
#' @param properties_file if not using the default credentials provider chain or |
|||
#' a named profile then provide the path to an Athena credentials proeprty file. |
|||
#' @param max maximum number of query execution ids to return. If `NULL` only the |
|||
#' most recent query execution ids will be returned. Otherwise sufficient |
|||
#' iterations of a call to the paginated API will be made to retrieve `max` |
|||
#' values. Note that the final total number of returned query execution ids |
|||
#' may be more or less than `max` depending on the number of paginated |
|||
#' results and no attempt will be made to truncate to `max`. |
|||
#' @export |
|||
list_query_executions <- function(region = "us-east-1", profile = NULL, properties_file = NULL, max = NULL) { |
|||
|
|||
client <- aws_athena_client(region = region, profile = profile, properties_file = properties_file) |
|||
|
|||
qx_req <- .jnew("com.amazonaws.services.athena.model.ListQueryExecutionsRequest") |
|||
|
|||
res <- client$listQueryExecutions(qx_req) |
|||
qxids <- res$getQueryExecutionIds() |
|||
out <- vapply(as.list(qxids), .jsimplify, character(1)) |
|||
|
|||
if (!is.null(max)) { |
|||
|
|||
repeat { |
|||
|
|||
if (max <= length(out)) break |
|||
if (is.jnull(res$getNextToken())) break |
|||
|
|||
res <- client$listQueryExecutions(qx_req$withNextToken(res$getNextToken())) |
|||
|
|||
qxids <- res$getQueryExecutionIds() |
|||
|
|||
out <- c(out, vapply(as.list(qxids), .jsimplify, character(1))) |
|||
|
|||
} |
|||
|
|||
} |
|||
|
|||
client$shutdown() |
|||
|
|||
out |
|||
|
|||
} |
@ -0,0 +1,16 @@ |
|||
#' @title rJava logging |
|||
#' @description Toggle verbose rJava logging |
|||
#' @note This resets a global Java setting and may affect logging of other rJava operations, |
|||
#' requiring a restart of R. |
|||
#' @return `NULL`, invisibly. |
|||
#' @author Thomas J. Leeper <thosjleeper@@gmail.com> |
|||
#' @noRd |
|||
#' @md |
|||
#' @examples |
|||
#' \dontrun{ |
|||
#' stop_logging() |
|||
#' } |
|||
stop_logging <- function() { |
|||
rJava::J("java.util.logging.LogManager")$getLogManager()$reset() |
|||
invisible(NULL) |
|||
} |
@ -0,0 +1,73 @@ |
|||
#' Download a key from a bucket to a local file |
|||
#' |
|||
#' @md |
|||
#' @param bucket,key S3 bucket and key (no `s3://` prefix) |
|||
#' @param output_dir where to store `key` |
|||
#' @param progress show download progress? |
|||
#' @param region AWS region string |
|||
#' @param profile if not using the default credentials chain or a dedicated |
|||
#' properties file then provide the named profile from `~/.aws/credentials` |
|||
#' you wish to use |
|||
#' @param properties_file if not using the default credentials provider chain or |
|||
#' a named profile then provide the path to an Athena credentials proeprty file. |
|||
#' @export |
|||
s3_download_file <- function(bucket, key, output_dir, |
|||
progress = FALSE, |
|||
region = "us-east-1", |
|||
profile = NULL, |
|||
properties_file = NULL) { |
|||
|
|||
aws_s3_client( |
|||
region = region, profile = profile, properties_file = properties_file |
|||
) -> client |
|||
|
|||
obj <- client$getObject(bucket, key) |
|||
meta <- obj$getObjectMetadata() |
|||
|
|||
total_size <- meta$getContentLength() |
|||
utils::txtProgressBar( |
|||
min = 0, max = total_size, initial = 0, |
|||
title = "S3 download progress", label = key |
|||
) -> pb |
|||
|
|||
outpath <- file.path(path.expand(output_dir), key) |
|||
|
|||
s3is <- obj$getObjectContent() |
|||
|
|||
buf <- raw(4096) |
|||
jbuf <- .jarray(buf) |
|||
|
|||
read_len <- s3is$read(jbuf) |
|||
|
|||
so_far <- 0 |
|||
|
|||
outfile <- file(outpath, open = "w+b") |
|||
|
|||
repeat { |
|||
|
|||
if (read_len <= 0) break |
|||
|
|||
outbuf <- .jevalArray(jbuf) |
|||
outbuf <- outbuf[1:read_len] |
|||
|
|||
writeBin(object = outbuf, con = outfile, size = 1, useBytes = TRUE) |
|||
|
|||
so_far <- so_far + read_len |
|||
|
|||
if (progress) utils::setTxtProgressBar(pb, so_far) |
|||
|
|||
read_len <- s3is$read(jbuf) |
|||
|
|||
} |
|||
|
|||
if (progress) close(pb) |
|||
|
|||
s3is$close() |
|||
|
|||
close(outfile) |
|||
|
|||
client$shutdown() |
|||
|
|||
invisible(outpath) |
|||
|
|||
} |
@ -0,0 +1,55 @@ |
|||
#' Start Query Execution |
|||
#' |
|||
#' @md |
|||
#' @param query SQL query statements to be executed |
|||
#' @param database database within which the query executes. |
|||
#' @param output_location location in S3 where query results are stored. |
|||
#' @param client_request_token unique case-sensitive string used to ensure the |
|||
#' request to create the query is idempotent (executes only once). If another |
|||
#' `StartQueryExecution` request is received, the same response is returned |
|||
#' and another query is not created. If a parameter has changed, for example, |
|||
#' the `query` , an error is returned. **This is auto-generated-for-you**. |
|||
#' @param encryption_option indicates whether Amazon S3 server-side encryption |
|||
#' with Amazon S3-managed keys (`SSE-S3`), server-side encryption with |
|||
#' KMS-managed keys (`SSE-KMS`), or client-side encryption with KMS-managed |
|||
#' keys (`CSE-KMS`) is used. Default is `NULL` (no encryption) |
|||
#' @param kms_key For `SSE-KMS` and `CSE-KMS`, this is the KMS key ARN or ID. |
|||
#' Default is `NULL` (no encryption) |
|||
#' @param region AWS region string |
|||
#' @param profile if not using the default credentials chain or a dedicated |
|||
#' properties file then provide the named profile from `~/.aws/credentials` |
|||
#' you wish to use |
|||
#' @param properties_file if not using the default credentials provider chain or |
|||
#' a named profile then provide the path to an Athena credentials proeprty file. |
|||
#' @export |
|||
start_query_execution <- function(query, database, output_location, |
|||
client_request_token = uuid::UUIDgenerate(), |
|||
encryption_option = NULL, |
|||
kms_key = NULL, |
|||
region = "us-east-1", |
|||
profile = NULL, |
|||
properties_file = NULL) { |
|||
|
|||
client <- aws_athena_client(region = region, profile = profile, properties_file = properties_file) |
|||
|
|||
ctx <- .jnew("com.amazonaws.services.athena.model.QueryExecutionContext") |
|||
ctx <- ctx$withDatabase(database) |
|||
|
|||
res_cfg <- .jnew("com.amazonaws.services.athena.model.ResultConfiguration") |
|||
res_cfg <- res_cfg$withOutputLocation(output_location) |
|||
|
|||
qx_req <- .jnew("com.amazonaws.services.athena.model.StartQueryExecutionRequest") |
|||
qx_req <- qx_req$withQueryString(query) |
|||
qx_req <- qx_req$withQueryExecutionContext(ctx) |
|||
qx_req <- qx_req$withResultConfiguration(res_cfg) |
|||
qx_req <- qx_req$withClientRequestToken(client_request_token) |
|||
|
|||
res <- client$startQueryExecution(qx_req) |
|||
|
|||
out <- res$getQueryExecutionId() |
|||
|
|||
client$shutdown() |
|||
|
|||
out |
|||
|
|||
} |
@ -0,0 +1,32 @@ |
|||
#' Stop Query Execution |
|||
#' |
|||
#' @md |
|||
#' @param qxid query execution id |
|||
#' @param region AWS region string |
|||
#' @param profile if not using the default credentials chain or a dedicated |
|||
#' properties file then provide the named profile from `~/.aws/credentials` |
|||
#' you wish to use |
|||
#' @param properties_file if not using the default credentials provider chain or |
|||
#' a named profile then provide the path to an Athena credentials proeprty file. |
|||
#' @export |
|||
stop_query_execution <- function(qxid, region = "us-east-1", profile = NULL, properties_file = NULL) { |
|||
|
|||
client <- aws_athena_client(region = region, profile = profile, properties_file = properties_file) |
|||
|
|||
qx_req <- .jnew("com.amazonaws.services.athena.model.StopQueryExecutionRequest") |
|||
qx_req <- qx_req$withQueryExecutionId(qxid) |
|||
|
|||
res <- client$stopQueryExecution(qx_req) |
|||
|
|||
qx_req <- .jnew("com.amazonaws.services.athena.model.GetQueryExecutionRequest") |
|||
qx_req <- qx_req$withQueryExecutionId(qxid) |
|||
|
|||
res <- client$getQueryExecution(qx_req) |
|||
|
|||
out <- res$getQueryExecution()$getStatus()$getState() |
|||
|
|||
client$shutdown() |
|||
|
|||
out |
|||
|
|||
} |
@ -0,0 +1,3 @@ |
|||
`%l0%` <- function(x, y) if (length(x) == 0) y else x |
|||
`%||%` <- function(x, y) if (is.null(x)) y else x |
|||
`%@%` <- function (x, name) attr(x, name, exact = TRUE) |
@ -0,0 +1,41 @@ |
|||
.properties_file_creds <- function(props_path) { |
|||
|
|||
props_path <- path.expand(props_path) |
|||
if (!file.exists(props_path)) { |
|||
stop(sprintf("%s file does not exist.", props_path), call.=FALSE) |
|||
} |
|||
|
|||
.jnew( |
|||
class = "com.amazonaws.auth.PropertiesFileCredentialsProvider", |
|||
props_path # credentialsFilePath |
|||
) -> properties_file_creds |
|||
|
|||
} |
|||
|
|||
.jawsregion <- function(reg = "us-east-1") { |
|||
regions <- J("com.amazonaws.regions.Regions") |
|||
reg <- toupper(gsub("-", "_", reg, fixed=TRUE)) |
|||
switch(reg, |
|||
"US_GOV_EAST_1" = regions$US_GOV_EAST_1, |
|||
"US_EAST_1" = regions$US_EAST_1, |
|||
"US_EAST_2" = regions$US_EAST_2, |
|||
"US_WEST_1" = regions$US_WEST_1, |
|||
"US_WEST_2" = regions$US_WEST_2, |
|||
"EU_WEST_1" = regions$EU_WEST_1, |
|||
"EU_WEST_2" = regions$EU_WEST_2, |
|||
"EU_WEST_3" = regions$EU_WEST_3, |
|||
"EU_CENTRAL_1" = regions$EU_CENTRAL_1, |
|||
"EU_NORTH_1" = regions$EU_NORTH_1, |
|||
"AP_SOUTH_1" = regions$AP_SOUTH_1, |
|||
"AP_SOUTHEAST_1" = regions$AP_SOUTHEAST_1, |
|||
"AP_SOUTHEAST_2" = regions$AP_SOUTHEAST_2, |
|||
"AP_NORTHEAST_1" = regions$AP_NORTHEAST_1, |
|||
"AP_NORTHEAST_2" = regions$AP_NORTHEAST_2, |
|||
"SA_EAST_1" = regions$SA_EAST_1, |
|||
"CN_NORTH_1" = regions$CN_NORTH_1, |
|||
"CN_NORTHWEST_1" = regions$CN_NORTHWEST_1, |
|||
"CA_CENTRAL_1" = regions$CA_CENTRAL_1, |
|||
"DEFAULT" = regions$DEFAULT_REGION, |
|||
stop("No such region.", call.=FALSE) |
|||
) |
|||
} |
@ -0,0 +1,8 @@ |
|||
.onAttach <- function(libname, pkgname) { |
|||
stop_logging() |
|||
} |
|||
|
|||
|
|||
.onLoad <- function(libname, pkgname){ |
|||
rJava::.jpackage(pkgname, lib.loc=libname) |
|||
} |
@ -0,0 +1,63 @@ |
|||
--- |
|||
output: rmarkdown::github_document |
|||
editor_options: |
|||
chunk_output_type: console |
|||
--- |
|||
|
|||
# awsathena |
|||
|
|||
rJava Interface to AWS Athena SDK |
|||
|
|||
## Description |
|||
|
|||
Provides R wrapper methods to core 'aws-java-sdk-athena' Java library methods |
|||
to interrogate and orchestrate Amazon Athena queries. |
|||
|
|||
## What's Inside the Tin? |
|||
|
|||
The following functions are implemented: |
|||
|
|||
- `collect_async`: Collect Amazon Athena 'dplyr' query results asynchronously |
|||
- `download_query_execution_results`: Use S3 to download the results of an Athena Query |
|||
- `get_query_execution`: Get Query Execution |
|||
- `list_query_executions`: List Query Executions |
|||
- `s3_download_file`: Download a key from a bucket to a local file |
|||
- `start_query_execution`: Start Query Execution |
|||
- `stop_query_execution`: Stop Query Execution |
|||
|
|||
## Installation |
|||
|
|||
```{r eval=FALSE} |
|||
devtools::install_git("git@git.sr.ht:~hrbrmstr/awsathena") |
|||
# OR |
|||
devtools::install_gitlab("hrbrmstr/awsathena") |
|||
# OR |
|||
devtools::install_github("hrbrmstr/awsathena") |
|||
``` |
|||
|
|||
## Usage |
|||
|
|||
```{r} |
|||
library(rJava) |
|||
library(awsathenajars) |
|||
library(awsathena) |
|||
|
|||
packageVersion("awsathena") |
|||
``` |
|||
|
|||
### Basics |
|||
|
|||
```{r} |
|||
qx <- list_query_executions(profile = "personal") |
|||
|
|||
head(qx) |
|||
|
|||
str(get_query_execution(qx[[1]], profile = "personal"), 1) |
|||
``` |
|||
|
|||
## `awsathenajars` Metrics |
|||
|
|||
```{r cloc, echo=FALSE} |
|||
cloc::cloc_pkg_md() |
|||
``` |
|||
|
@ -0,0 +1,83 @@ |
|||
|
|||
# awsathena |
|||
|
|||
rJava Interface to AWS Athena SDK |
|||
|
|||
## Description |
|||
|
|||
Provides R wrapper methods to core ‘aws-java-sdk-athena’ Java library |
|||
methods to interrogate and orchestrate Amazon Athena queries. |
|||
|
|||
## What’s Inside the Tin? |
|||
|
|||
The following functions are implemented: |
|||
|
|||
- `collect_async`: Collect Amazon Athena ‘dplyr’ query results |
|||
asynchronously |
|||
- `download_query_execution_results`: Use S3 to download the results |
|||
of an Athena Query |
|||
- `get_query_execution`: Get Query Execution |
|||
- `list_query_executions`: List Query Executions |
|||
- `s3_download_file`: Download a key from a bucket to a local file |
|||
- `start_query_execution`: Start Query Execution |
|||
- `stop_query_execution`: Stop Query Execution |
|||
|
|||
## Installation |
|||
|
|||
``` r |
|||
devtools::install_git("git@git.sr.ht:~hrbrmstr/awsathena") |
|||
# OR |
|||
devtools::install_gitlab("hrbrmstr/awsathena") |
|||
# OR |
|||
devtools::install_github("hrbrmstr/awsathena") |
|||
``` |
|||
|
|||
## Usage |
|||
|
|||
``` r |
|||
library(rJava) |
|||
library(awsathenajars) |
|||
library(awsathena) |
|||
|
|||
packageVersion("awsathena") |
|||
``` |
|||
|
|||
## [1] '0.1.0' |
|||
|
|||
### Basics |
|||
|
|||
``` r |
|||
qx <- list_query_executions(profile = "personal") |
|||
|
|||
head(qx) |
|||
``` |
|||
|
|||
## [1] "d287b857-5801-493e-a647-76b5cacdac71" |
|||
## [2] "cbaad9f0-9231-443b-82dc-4cd8c8fad21b" |
|||
## [3] "8e5d155f-4eda-44fa-b2a4-2245d71b7f41" |
|||
## [4] "fb1d48d3-6875-4681-b01e-c59a5a33377d" |
|||
## [5] "cb7d50fb-a2c7-4ada-a4f9-ea35fbbaf764" |
|||
## [6] "5ddc2339-1987-4890-a521-1cee09b1e291" |
|||
|
|||
``` r |
|||
str(get_query_execution(qx[[1]], profile = "personal"), 1) |
|||
``` |
|||
|
|||
## List of 10 |
|||
## $ workgroup : chr "primary" |
|||
## $ query : chr "SELECT \"tsday\", \"host\", \"receivedbytes\", \"requestprocessingtime\", \"proto_version\"\nFROM (SELECT \"tim"| __truncated__ |
|||
## $ context : chr "sampledb" |
|||
## $ statement_type : chr "DML" |
|||
## $ state : chr "SUCCEEDED" |
|||
## $ started : chr "19 Feb 2019 19:11:51 GMT" |
|||
## $ completed : chr "19 Feb 2019 19:11:52 GMT" |
|||
## $ output_location: chr "s3://aws-athena-query-results-569593279821-us-east-1/d287b857-5801-493e-a647-76b5cacdac71.csv" |
|||
## $ data_scanned : num 351173 |
|||
## $ execution_time : num 1142 |
|||
|
|||
## `awsathenajars` Metrics |
|||
|
|||
| Lang | \# Files | (%) | LoC | (%) | Blank lines | (%) | \# Lines | (%) | |
|||
| :--- | -------: | ---: | --: | ---: | ----------: | ---: | -------: | ---: | |
|||
| R | 15 | 0.94 | 255 | 0.96 | 101 | 0.84 | 180 | 0.85 | |
|||
| Rmd | 1 | 0.06 | 11 | 0.04 | 19 | 0.16 | 33 | 0.15 | |
@ -0,0 +1,21 @@ |
|||
Version: 1.0 |
|||
|
|||
RestoreWorkspace: Default |
|||
SaveWorkspace: Default |
|||
AlwaysSaveHistory: Default |
|||
|
|||
EnableCodeIndexing: Yes |
|||
UseSpacesForTab: Yes |
|||
NumSpacesForTab: 2 |
|||
Encoding: UTF-8 |
|||
|
|||
RnwWeave: Sweave |
|||
LaTeX: pdfLaTeX |
|||
|
|||
StripTrailingWhitespace: Yes |
|||
|
|||
BuildType: Package |
|||
PackageUseDevtools: Yes |
|||
PackageInstallArgs: --no-multiarch --with-keep.source |
|||
PackageBuildArgs: --resave-data |
|||
PackageRoxygenize: rd,collate,namespace |
@ -0,0 +1,15 @@ |
|||
% Generated by roxygen2: do not edit by hand |
|||
% Please edit documentation in R/awsathena-package.R |
|||
\docType{package} |
|||
\name{awsathena} |
|||
\alias{awsathena} |
|||
\alias{awsathena-package} |
|||
\title{rJava Client to AWS Athena SDK} |
|||
\description{ |
|||
Provides R wrapper methods to core 'aws-java-sdk-athena' Java library methods |
|||
to interrogate and orchestrate Amazon Athena queries. |
|||
} |
|||
\author{ |
|||
Bob Rudis (bob@rud.is) |
|||
} |
|||
\keyword{internal} |
@ -0,0 +1,87 @@ |
|||
% Generated by roxygen2: do not edit by hand |
|||
% Please edit documentation in R/collect-async.R |
|||
\name{collect_async} |
|||
\alias{collect_async} |
|||
\title{Collect Amazon Athena \code{dplyr} query results asynchronously} |
|||
\usage{ |
|||
collect_async(obj, database, output_location, |
|||
client_request_token = uuid::UUIDgenerate(), |
|||
encryption_option = NULL, kms_key = NULL, region = "us-east-1", |
|||
profile = NULL, properties_file = NULL) |
|||
} |
|||
\arguments{ |
|||
\item{obj}{the \code{dplyr} query} |
|||
|
|||
\item{database}{database within which the query executes.} |
|||
|
|||
\item{output_location}{location in S3 where query results are stored.} |
|||
|
|||
\item{client_request_token}{unique case-sensitive string used to ensure the |
|||
request to create the query is idempotent (executes only once). If another |
|||
\code{StartQueryExecution} request is received, the same response is returned |
|||
and another query is not created. If a parameter has changed, for example, |
|||
the \code{query} , an error is returned. \strong{This is auto-generated-for-you}.} |
|||
|
|||
\item{encryption_option}{indicates whether Amazon S3 server-side encryption |
|||
with Amazon S3-managed keys (\code{SSE-S3}), server-side encryption with |
|||
KMS-managed keys (\code{SSE-KMS}), or client-side encryption with KMS-managed |
|||
keys (\code{CSE-KMS}) is used. Default is \code{NULL} (no encryption)} |
|||
|
|||
\item{kms_key}{For \code{SSE-KMS} and \code{CSE-KMS}, this is the KMS key ARN or ID. |
|||
Default is \code{NULL} (no encryption)} |
|||
|
|||
\item{region}{AWS region string} |
|||
|
|||
\item{profile}{if not using the default credentials chain or a dedicated |
|||
properties file then provide the named profile from \code{~/.aws/credentials} |
|||
you wish to use} |
|||
|
|||
\item{properties_file}{if not using the default credentials provider chain or |
|||
a named profile then provide the path to an Athena credentials proeprty file.} |
|||
} |
|||
\description{ |
|||
Long running Athena queries and Athena queries with large result |
|||
sets can seriously stall a \code{dplyr} processing chain due to poorly |
|||
implemented ODBC and JDBC drivers. This function converts a \code{dplyr} |
|||
chain to a raw SQL query then submits it via \code{\link[=start_query_execution]{start_query_execution()}} and |
|||
returns the query execution id. You can retrieve the result set either |
|||
via \code{\link[=download_query_execution_results]{download_query_execution_results()}} or your favorite S3 downloading |
|||
method. |
|||
} |
|||
\note{ |
|||
\code{dbplyr} must be installed for this to work. It is not listed in |
|||
the \code{Imports} as it brings with it many dependencies that may not |
|||
be necessary for general use of this package. |
|||
} |
|||
\examples{ |
|||
\dontrun{ |
|||
library(odbc) |
|||
library(DBI) |
|||
|
|||
DBI::dbConnect( |
|||
odbc::odbc(), |
|||
driver = "/Library/simba/athenaodbc/lib/libathenaodbc_sbu.dylib", |
|||
Schema = "sampledb", |
|||
AwsRegion = "us-east-1", |
|||
AwsProfile = "personal", |
|||
AuthenticationType = "IAM Profile", |
|||
S3OutputLocation = "s3://aws-athena-query-results-redacted" |
|||
) -> con |
|||
|
|||
elb_logs <- tbl(con, "elb_logs") |
|||
|
|||
mutate(elb_logs, tsday = substr(timestamp, 1, 10)) \%>\% |
|||
filter(tsday == "2014-09-29") \%>\% |
|||
select(requestip, requestprocessingtime) \%>\% |
|||
collect_async( |
|||
database = "sampledb", |
|||
output_location = "s3://aws-athena-query-results-redacted", |
|||
profile = "personal" |
|||
) -> id |
|||
|
|||
get_query_execution(id, profile = "personal") |
|||
|
|||
# do this to get the data or use your favorite way to grab files from S3 |
|||
get_query_results(id, profile = "personal") |
|||
} |
|||
} |
@ -0,0 +1,29 @@ |
|||
% Generated by roxygen2: do not edit by hand |
|||
% Please edit documentation in R/download-query-ex-res.R |
|||
\name{download_query_execution_results} |
|||
\alias{download_query_execution_results} |
|||
\title{Use S3 to download the results of an Athena Query} |
|||
\usage{ |
|||
download_query_execution_results(qxid, output_dir, progress = FALSE, |
|||
region = "us-east-1", profile = NULL, properties_file = NULL) |
|||
} |
|||
\arguments{ |
|||
\item{qxid}{query execution id} |
|||
|
|||
\item{output_dir}{either a full or relative path (\code{\link[=path.expand]{path.expand()}} will be run on |
|||
the value) to where you want the results to be stored.} |
|||
|
|||
\item{progress}{show download progress?} |
|||
|
|||
\item{region}{AWS region string} |
|||
|
|||
\item{profile}{if not using the default credentials chain or a dedicated |
|||
properties file then provide the named profile from \code{~/.aws/credentials} |
|||
you wish to use} |
|||
|
|||
\item{properties_file}{if not using the default credentials provider chain or |
|||
a named profile then provide the path to an Athena credentials proeprty file.} |
|||
} |
|||
\description{ |
|||
This method is much faster than the paginated interface |
|||
} |
@ -0,0 +1,24 @@ |
|||
% Generated by roxygen2: do not edit by hand |
|||
% Please edit documentation in R/get-query-execution.R |
|||
\name{get_query_execution} |
|||
\alias{get_query_execution} |
|||
\title{Get Query Execution} |
|||
\usage{ |
|||
get_query_execution(qxid, region = "us-east-1", profile = NULL, |
|||
properties_file = NULL) |
|||
} |
|||
\arguments{ |
|||
\item{qxid}{query execution id} |
|||
|
|||
\item{region}{AWS region string} |
|||
|
|||
\item{profile}{if not using the default credentials chain or a dedicated |
|||
properties file then provide the named profile from \code{~/.aws/credentials} |
|||
you wish to use} |
|||
|
|||
\item{properties_file}{if not using the default credentials provider chain or |
|||
a named profile then provide the path to an Athena credentials proeprty file.} |
|||
} |
|||
\description{ |
|||
Get Query Execution |
|||
} |
@ -0,0 +1,29 @@ |
|||
% Generated by roxygen2: do not edit by hand |
|||
% Please edit documentation in R/list-query-executions.R |
|||
\name{list_query_executions} |
|||
\alias{list_query_executions} |
|||
\title{List Query Executions} |
|||
\usage{ |
|||
list_query_executions(region = "us-east-1", profile = NULL, |
|||
properties_file = NULL, max = NULL) |
|||
} |
|||
\arguments{ |
|||
\item{region}{AWS region string} |
|||
|
|||
\item{profile}{if not using the default credentials chain or a dedicated |
|||
properties file then provide the named profile from \code{~/.aws/credentials} |
|||
you wish to use} |
|||
|
|||
\item{properties_file}{if not using the default credentials provider chain or |
|||
a named profile then provide the path to an Athena credentials proeprty file.} |
|||
|
|||
\item{max}{maximum number of query execution ids to return. If \code{NULL} only the |
|||
most recent query execution ids will be returned. Otherwise sufficient |
|||
iterations of a call to the paginated API will be made to retrieve \code{max} |
|||
values. Note that the final total number of returned query execution ids |
|||
may be more or less than \code{max} depending on the number of paginated |
|||
results and no attempt will be made to truncate to \code{max}.} |
|||
} |
|||
\description{ |
|||
List Query Executions |
|||
} |
@ -0,0 +1,28 @@ |
|||
% Generated by roxygen2: do not edit by hand |
|||
% Please edit documentation in R/s3-download-file.R |
|||
\name{s3_download_file} |
|||
\alias{s3_download_file} |
|||
\title{Download a key from a bucket to a local file} |
|||
\usage{ |
|||
s3_download_file(bucket, key, output_dir, progress = FALSE, |
|||
region = "us-east-1", profile = NULL, properties_file = NULL) |
|||
} |
|||
\arguments{ |
|||
\item{bucket, key}{S3 bucket and key (no \code{s3://} prefix)} |
|||
|
|||
\item{output_dir}{where to store \code{key}} |
|||
|
|||
\item{progress}{show download progress?} |
|||
|
|||
\item{region}{AWS region string} |
|||
|
|||
\item{profile}{if not using the default credentials chain or a dedicated |
|||
properties file then provide the named profile from \code{~/.aws/credentials} |
|||
you wish to use} |
|||
|
|||
\item{properties_file}{if not using the default credentials provider chain or |
|||
a named profile then provide the path to an Athena credentials proeprty file.} |
|||
} |
|||
\description{ |
|||
Download a key from a bucket to a local file |
|||
} |
@ -0,0 +1,44 @@ |
|||
% Generated by roxygen2: do not edit by hand |
|||
% Please edit documentation in R/start-query-execution.R |
|||
\name{start_query_execution} |
|||
\alias{start_query_execution} |
|||
\title{Start Query Execution} |
|||
\usage{ |
|||
start_query_execution(query, database, output_location, |
|||
client_request_token = uuid::UUIDgenerate(), |
|||
encryption_option = NULL, kms_key = NULL, region = "us-east-1", |
|||
profile = NULL, properties_file = NULL) |
|||
} |
|||
\arguments{ |
|||
\item{query}{SQL query statements to be executed} |
|||
|
|||
\item{database}{database within which the query executes.} |
|||
|
|||
\item{output_location}{location in S3 where query results are stored.} |
|||
|
|||
\item{client_request_token}{unique case-sensitive string used to ensure the |
|||
request to create the query is idempotent (executes only once). If another |
|||
\code{StartQueryExecution} request is received, the same response is returned |
|||
and another query is not created. If a parameter has changed, for example, |
|||
the \code{query} , an error is returned. \strong{This is auto-generated-for-you}.} |
|||
|
|||
\item{encryption_option}{indicates whether Amazon S3 server-side encryption |
|||
with Amazon S3-managed keys (\code{SSE-S3}), server-side encryption with |
|||
KMS-managed keys (\code{SSE-KMS}), or client-side encryption with KMS-managed |
|||
keys (\code{CSE-KMS}) is used. Default is \code{NULL} (no encryption)} |
|||
|
|||
\item{kms_key}{For \code{SSE-KMS} and \code{CSE-KMS}, this is the KMS key ARN or ID. |
|||
Default is \code{NULL} (no encryption)} |
|||
|
|||
\item{region}{AWS region string} |
|||
|
|||
\item{profile}{if not using the default credentials chain or a dedicated |
|||
properties file then provide the named profile from \code{~/.aws/credentials} |
|||
you wish to use} |
|||
|
|||
\item{properties_file}{if not using the default credentials provider chain or |
|||
a named profile then provide the path to an Athena credentials proeprty file.} |
|||
} |
|||
\description{ |
|||
Start Query Execution |
|||
} |
@ -0,0 +1,24 @@ |
|||
% Generated by roxygen2: do not edit by hand |
|||
% Please edit documentation in R/stop-query-execution.R |
|||
\name{stop_query_execution} |
|||
\alias{stop_query_execution} |
|||
\title{Stop Query Execution} |
|||
\usage{ |
|||
stop_query_execution(qxid, region = "us-east-1", profile = NULL, |
|||
properties_file = NULL) |
|||
} |
|||
\arguments{ |
|||
\item{qxid}{query execution id} |
|||
|
|||
\item{region}{AWS region string} |
|||
|
|||
\item{profile}{if not using the default credentials chain or a dedicated |
|||
properties file then provide the named profile from \code{~/.aws/credentials} |
|||
you wish to use} |
|||
|
|||
\item{properties_file}{if not using the default credentials provider chain or |
|||
a named profile then provide the path to an Athena credentials proeprty file.} |
|||
} |
|||
\description{ |
|||
Stop Query Execution |
|||
} |
@ -0,0 +1,2 @@ |
|||
library(testthat) |
|||
test_check("awsathena") |
@ -0,0 +1,4 @@ |
|||
context("basic functionality") |
|||
test_that("we can do something", { |
|||
|
|||
}) |
Loading…
Reference in new issue