Add dbWriteTable method for the CSV

[artemklevtsov]

To load CSV directly:

tmp <- tempfile(fileext = ".csv") utils::write.csv(iris, tmp) DBI::dbWriteTable( conn = conn, name = "iris", value = tmp, temporary = TRUE )

[krlmlr]

Thanks! Can you comment if this approach could be a viable solution for the performance problems in RMariaDB too:

• LOAD DATA LOCAL INFILE vs INSERT INTO RMariaDB#11
• Slow writing data to MySQL database hosted on Google Cloud RMariaDB#104
• dbWriteTable() much slower than RMySQL, especially slow on Mac RMariaDB#125
• Perf issue with dbWriteTable : RMariaDB much slower than RMySQL (from 2 to 40 times), and it's worse with a remote db RMariaDB#162

[artemklevtsov]

I think the LOAD LOCAL DATA INFILE statement should solve the performance issues. Far as I can see, this has already been implemented. Didn't that solve the problem?

My goal in this PR was just to reduce memory usage for more optimal ETL.

[krlmlr]

Thanks. Instead of adding yet another dbWriteTable() method, I'd rather implement a new postgresAppendTableCSV() that does everything minus table creation/removal. We can later decide if we add a new dbAppendTable() method or implement a new dbAppendTableCSV() generic.

[krlmlr]

Would you like to help with a postgresAppendTableCSV() method?

[artemklevtsov]

Sure. Can you provide more details? I didn't find anything in this repo.

[krlmlr]

DBI now has dbCreateTable() and dbAppendTable(). If we want to implement a new method for writing CSV files, we should implement postgresAppendTableCSV() -- without duplicationg the logic of creating tables. Later we could think about providing a dbAppendTable() method that forwards to postgresAppendTableCSV().

[artemklevtsov]

Fast draft the postgresAppendTableCSV may be looks like:

postgresAppendTableCSV <- function(conn, name, value, fields = NULL, sep = ",", header = TRUE, na.strings = "NA", encoding = "UTF-8") { sql <- paste0( "COPY ", dbQuoteIdentifier(conn, name), " (", paste(dbQuoteIdentifier(conn, fields), collapse = ","), ") ", "FROM STDIN ", "(FORMAT CSV DELIMITER '", sep, "', HEADER '", header, "' NULL '", na.strings, "' ENCODING '", encoding, "')" ) connection_copy_file(conn@ptr, sql, path.expand(value)) invisible(TRUE) }