sql_fun,DuckDBTable-method built its return_type IN (...) predicate by
pasting the user-supplied return_type argument directly between
hardcoded quotes (paste0("'", return_type, "'", ...)), with no
escaping, before sending the query to dbGetQuery(). return_type was
validated only as is.character(), so a value containing a ' (e.g.
"x') OR 1=1 --") broke out of the string literal. Real-world
exploitability was low (this queries DuckDB's own local, in-process,
read-only duckdb_functions() catalog inside an embedded single-user
database, no network boundary or cross-tenant data involved), but it was
the one place in the file not using DBI::dbQuoteString() despite DBI
already being imported in the same method. The parallel
function_type IN (...) predicate built the same way was not actually
exploitable (match.arg(..., several.ok = TRUE) constrains it to a
fixed enum first) but is fixed the same way for consistency. Both now
quote via DBI::dbQuoteString(conn, ...).Reverted
is.na() and anyNA() on a DuckDBColumn, DuckDBTable, or
DuckDBDataFrame silently reported no missing values at all, regardless
of the actual data. Neither class defined its own is.na/anyNA, so both
inherited S4Vectors' Vector/List fallback stubs
(rep.int(FALSE, length(x)) and function(x, recursive = FALSE) FALSE),
which exist for abstract Vector subclasses with no inherent notion of
missingness -- the wrong inheritance for a class backed by real,
well-defined per-row SQL NULL semantics. Both now have real methods:
is.na() delegates to sql_call(x, "is.na") (dbplyr's SQL translator
already renders the R function name is.na as IS NULL, so no new
query-building code was needed, only reusing the existing is.finite/
is.infinite/is.nan delegation pattern with a different function name);
anyNA() ORs every data column's "is missing" expression into one per-row
predicate and evaluates a single any() aggregate of it in one query.
Because DuckDBDataFrame extends DuckDBTable before DataFrame,
anyNA() needed to work correctly for any number of data columns itself
rather than relying on the S4Vectors per-column DataFrame composition,
which is never reached.is.finite/
is.infinite/is.nan test to a special_path fixture that already
includes a NaN: DuckDB's x IS NULL does not treat NaN as missing
(NaN IS NULL is FALSE in DuckDB), but R's is.na() does (is.na(NaN)
is TRUE). is.na()/anyNA() now additionally check isnan(x) for
"double"-typed columns specifically, matching R's semantics exactly
without affecting any other column type.Opening a directory required every file in it to be parquet, so a
resource directory carrying an ordinary sibling (_SUCCESS,
_common_metadata, a README) was left unwrapped and then resolved as a
catalog table name, failing with a leaked SQL parser error quoting the
path. The test was also written any(all(...)), which is just all(...):
all() already returns a single logical. A directory is now opened when it
contains any parquet part, and the glob targets the parquet extensions
actually present rather than **, which would otherwise hand the siblings
to read_parquet().
DuckDBDataFrame() now refuses a directory carrying an _INCOMPLETE
marker, naming it in the error. This has to ship together with the change
above: refusing such a directory was previously only a side effect of
requiring every file to be parquet, so relaxing that test on its own would
have made an interrupted multi-part write silently readable as partial
data (verified: it returns the parts written so far, with no error).
_INCOMPLETE is written by BiocDuckDB's writeStreamingResource() for the
duration of a stream.
splitParquetPart() could destroy the data it was splitting. It called
unlink() on the source file before copying the new parts into place,
discarded file.copy()'s return value, and left overwrite = FALSE, so a
copy that failed for any reason (a full disk, a permission or filesystem
error) removed the only copy of the data and then returned without
signalling an error, leaving a manifest pointing at a resource that no
longer existed. The source file is now moved aside rather than deleted, the
copy's success is checked, and any failure rolls the directory back to
exactly its original contents before erroring. The move-aside is needed
because a newly written part can legitimately have the same name as the
source file; it stays within the resource directory so it remains a
same-filesystem rename.
writeDuckDBTableParquet() dropped factor columns' levels. The lazy write
path is a SQL COPY, and DuckDB has no categorical type, so every factor
was flattened to plain VARCHAR; any level unused in the data was lost
outright, since it is not recoverable from the written values alone. The
recorded collevels were never consulted, even though the equivalent
arrow-side fixup already existed in this file for splitParquetPart()'s
own DuckDB round trip. That fixup is now factored into a shared internal
helper and applied on this path too, before the returned sample_df is
read back, so the caller's schema inference also sees the factors and can
emit the categories/categoriesOrdered schema keywords correctly.
Non-factor columns are untouched.
The factor-restoration path no longer leaves the parquet files it inspects
open, which made it fail on Windows. It rewrites each file in place, and
splitParquetPart() additionally renames and then unlinks its source file;
Windows refuses all three operations while any mapping or handle on the
file is still live. Two distinct leaks were involved:
arrow's default memory-mapped read ([Windows error 1224] The requested operation cannot be performed on a file with a user-mapped section open). All reads on this path now pass mmap = FALSE. Staging
the write elsewhere and renaming into place would not have helped, since
replacing a mapped file is blocked in the same way..findFactorColumns() needs only the schema, but
open_dataset() cannot opt out of memory-mapping and ParquetFileReader
has no Close() method, so either one leaves the file open until the
reader is garbage-collected. It now creates its own ReadableFile,
passes that to ParquetFileReader, and closes it explicitly; the
returned Schema remains valid afterwards.splitParquetPart() also drops its DuckDB temporary table before touching
the source file, rather than at function exit, so nothing on the DuckDB
side still refers to it either.
The schema read is now shared by readParquetSchema() and
.findFactorColumns() through one internal helper. readParquetSchema()
had the same leak (verified: one open handle per call, now none). It was
not reachable as a bug, since its only caller appends a new part rather
than rewriting the file it inspected, but it is exported and one step away
from the paths that do rewrite. It also no longer reads the file's data
just to obtain its schema.
This affected the pre-existing splitParquetPart() restoration path as
well as the new one; it surfaced only now because that function had no
test coverage before this release.
coltypes()/type() (and the show() header row they drive) now report
"factor" for a character column with a recorded collevels entry,
instead of "character". collevels (set by readParquet() from the
schema's categories/categoriesOrdered, a standard Frictionless Table
Schema keyword) already correctly restored factor columns on
materialization (as.data.frame(), as.vector()); only the
non-materializing type-reporting methods hadn't been taught to look at
it, so a lazy DuckDBDataFrame's colData() display showed
<character> for a column that was, in fact, a factor as soon as you
pulled it out. coltypes<- is unchanged: "factor" is not a settable
cast target, and recasting a factor column's underlying type still
drops its collevels entry as before.levels()/nlevels() methods for DuckDBColumn, returning the
recorded collevels levels (or NULL/0) without materializing the
column.splitParquetPart() was silently dropping factor columns to plain
character, and losing any factor level unused in the data outright
(unrecoverable from the written values alone), because DuckDB has no
categorical/dictionary SQL type and flattens an Arrow dictionary-encoded
column to VARCHAR. Factor columns (including ordered ones and unused
levels) are now detected from the source file's schema before the
DuckDB-based row split and restored on each output part afterward via a
small, per-part arrow-only read/rewrite.splitParquetPart(), a parquet-io helper that rebalances a
directory's single flat part-*.parquet file into n_parts smaller
ones, in place, e.g. after a writer emitted everything into a single
large part-0.parquet. Row order is preserved: part 0 gets the first
~1/n_parts rows in on-disk order, part 1 the next ~1/n_parts, and so
on, as if the file had simply been cut into contiguous slices. Uses
DuckDB's file_row_number Parquet-reader option and an explicit
ORDER BY on both the bucket assignment and each part's write, since
neither a bare ntile() OVER () nor an unordered read-back is
guaranteed to match on-disk order under DuckDB's parallel scan.@aliases coerce,DuckDBSelfHits,SelfHits-method entry
in ?DuckDBSelfHits-class; the setAs("DuckDBSelfHits", "SelfHits", ...)
coercion was implemented but not documented, which R CMD check flagged
as a missing documentation entry..-prefixed names should not be exported:
.keycols -> keycols, .has_row_number -> has_row_number,
.set_row_number -> set_row_number, and
.makePrettyCharacterMatrixForDisplay -> makePrettyCharacterMatrixForDisplay.
These remain the shared cross-package extension API and are now documented on a
dedicated internal man page (?DuckDBDataFrame-internals, marked
\keyword{internal}). Companion packages (DuckDBGRanges, and callers in
BiocDuckDB) are updated to the new names.mpg > 25) rather than
trailing "predicate pushdown, direct Parquet scans".URL and BugReports fields to DESCRIPTION.?DuckDBDataFrame).sql_call() and sql_fun().datacols/keycol are optional) and the
keycol = "model" versus keycol = list(model = ...) forms; clarified that
adding a computed column changes only the in-memory object (a lazy SQL
expression) and does not modify the Parquet file; changed the sql_call()
rounding example to round(., 0).DuckDBTable with a
keyless DuckDBDataFrame (identical() FALSE, all.equal() TRUE) and added
.has_row_number() under key-dimension semantics; clarified how dimension
tables enable partition pruning; added a minimal example of extending
DuckDBDataFrame.library(BiocStyle) call from both vignettes..get_type().clusterOrderSQL() and columnExtents(), the SQL-side primitives
behind the cluster_by write path. clusterOrderSQL(conn, subquery_sql, cluster_by, available) lowers a zorder() / hilbert() / character spec into
ORDER BY expressions for buildParquetCopySQL(order_cols=); columnExtents()
returns per-column finite [min, max] over a subquery. These let another
package (e.g. DuckDBArray's coord-array writer, and a future serving-layout
builder) cluster a COPY TO without re-deriving the Morton / Hilbert generator,
keeping a single curve implementation across the suite. No behavior change to
existing exports; this only surfaces internals as a documented API.writeDuckDBTableParquet() no longer drops the row-group size. It gains a
row_group_size argument defaulting to 491520L (240 x the 2048 DuckDB
STANDARD_VECTOR_SIZE) and threads it into buildParquetCopySQL(); previously
the lazy DuckDBTable export passed no size and fell back to DuckDB's 122880
default while every other writer path (coord arrays, flat arrow) already
targeted 491520. This affected the wide colData table and lazy-written graph
edge lists. Faithful repack (row-group count only); the data is unchanged.show(<DuckDBSelfHits>) now format()s the hit and node counts before printing
them, so a graph with more than 2^31 hits (or nodes) reports the true count
instead of a denormalized double. nrow() returns a bit64 integer64 once the
count exceeds .Machine$integer.max, and cat() prints an integer64's
underlying bit pattern (e.g. 1.377174e-314) rather than its value; a k=100
neighbor graph over ~28M cells (~2.78e9 edges) tripped this, while a sub-2^31
graph printed correctly. Display-only: the edge data was always correct.as.matrix(<DuckDBEmbeddings>) now unname()s the row keys before assigning
them, so an aliased keycol no longer leaves a stray names attribute on the
matrix's dimnames[[1]]. .map_keycol_names() returns a NAMED character
vector for an aliased key, and unlike names<-, rownames<-/dimnames<-
keeps that inner attribute — which broke equality against a plain-rownames
matrix (e.g. a reduced-dimension round-trip). Regression from the 0.99.7
query-order label fix.memory_limit / threads detectors used by configureOutOfCore()
now resolve the job's own cgroup subpath from /proc/self/cgroup (both
cgroup v2 unified and v1 per-controller, including combined mounts like
cpu,cpuacct) before falling back to the mount root. A cgroup-constrained
non-SLURM job was previously undetected — the fixed root paths
(/sys/fs/cgroup/memory.max, …) only match when the process runs in the root
cgroup — so it silently fell back to physical RAM / hardware concurrency, the
over-commit the defaults are meant to prevent. The mount root stays in the
search list, so detection never regresses where it already worked. The
detectors (.cgroupDirs(), .cgroupMemoryBytes(), .cgroupCpus(),
.physicalMemoryBytes()) take injectable path parameters and are now covered
by fixture-based unit tests.configureOutOfCore() now also defaults DuckDB threads to the job's CPU
allocation when neither DuckDBDataFrame.threads nor BIOCDUCKDB_THREADS is
set: the most-restrictive of a SLURM allocation (SLURM_CPUS_PER_TASK, or
SLURM_CPUS_ON_NODE) and the cgroup CFS quota (v2 cpu.max, v1
cpu.cfs_quota_us/cpu.cfs_period_us). DuckDB otherwise defaults threads to
hardware concurrency, which ignores a SLURM/cgroup cpuset and over-subscribes
on a shared node — and each extra thread's working memory pushes against the
same cgroup the memory_limit default (0.99.10) targets. When no allocation
is detected, DuckDB's default is left in place. New internal helpers
.slurmCpus() / .cgroupCpus() / .defaultThreads().as.vector(<DuckDBColumn>), as.list(<DuckDBAtomicList>), and
as.data.frame(<DuckDBDataFrame>) now reorder their query result into the
object's canonical stored-key order length-safely, extending the
as.matrix(<DuckDBEmbeddings>) fix (0.99.7) to its siblings. They previously
re-indexed by the stored key vector (result[rownames(x@table)]) after
labelling from the materialized keycol; when the stored key set diverged from
the materialized rows (a subset filtered through a dimension table, or
aliased / duplicate keys) that silently NA-padded, first-matched a duplicate,
or truncated. The reorder now happens only when the stored keys and
materialized names are a clean 1:1 correspondence and otherwise falls back to
query order with every value kept paired with its true name; a row_number
key keeps query order (and no longer triggers a full key scan just to
reorder). New internal helpers .storedKeysBijective() / .reindexByStoredKeys().configureOutOfCore() now defaults the DuckDB memory_limit to 80% of the
most-restrictive detected ceiling when neither DuckDBDataFrame.memory_limit
nor BIOCDUCKDB_MEMORY_LIMIT is set: an explicit SLURM allocation
(SLURM_MEM_PER_NODE, or SLURM_MEM_PER_CPU times SLURM_CPUS_ON_NODE), the
cgroup limit (v2 memory.max then v1 memory.limit_in_bytes), then physical
RAM (/proc/meminfo). DuckDB's own default is 80% of physical RAM, which
ignores a SLURM / cgroup cap and over-commits — nearly OOM-ing on the first
large scan of a big out-of-core aggregation instead of spilling. When nothing
can be detected (e.g. macOS) DuckDB's default is left in place.configureOutOfCore() now always sets and creates (recursively) the DuckDB
spill temp_directory. When neither DuckDBDataFrame.temp_directory nor
BIOCDUCKDB_TEMP_DIRECTORY is set it defaults to a temp subdirectory of
R_user_dir("DuckDBDataFrame", "cache") instead of relying on DuckDB's
default under the R session tempdir, and a configured path is created before
SET temp_directory. On batch schedulers whose per-job tempdir (e.g. SLURM
/tmp) is small or cleaned mid-session, a large out-of-core sort/aggregation
previously failed with "IO Error: Failed to create directory ... No such file
or directory"; the spill directory is now guaranteed to exist.as.matrix methods table from S4Vectors to ensure the methods
defined in this package are exported.as.matrix(<DuckDBEmbeddings>) now derives row names from the same
materialized query as the matrix (the keycol column) rather than from the
stored keycol slot via rownames(). For a named key, rownames() returns
keydimnames() = the stored keycols vector, whose length can diverge from
the materialized row count (e.g. a subset filtered through a dimtbl, or
aliased / duplicate keys). That divergence made show()/as.matrix() on a
large embedding fail with "length of 'dimnames' [1] not equal to array
extent". Row names are now consistent with the row count by construction.buildParquetCopySQL() gains an append argument that emits the DuckDB
APPEND copy option. This lets a PARTITION_BY write add new files to an
existing (hive-partitioned) directory instead of failing "directory is not
empty", which is what a coord-array append needs (DuckDBArray).writeDuckDBTableParquet() / buildTableSelectSQL())
now types the __index__ column the same way the in-memory writer does,
instead of always emitting a BIGINT row_number(). It CASTs the index to a
type chosen by range (narrowed on a fresh write, index_max honored, or pinned
to part 0's on-disk type on append), so a resource written or appended across
both write paths keeps one consistent __index__ type. Previously an
in-memory part 0 (narrowed) plus a lazy append (BIGINT) produced a
schema-inconsistent, unreadable resource, and the same table had a different
index type depending on which path wrote it.DuckDBSelfHits() now fails loudly when nnode (or a supplied node id)
exceeds the 32-bit integer range, instead of letting as.integer() silently
coerce it to NA (which corrupted graph reconstruction). Graphs with more
than ~2.1e9 nodes are not yet supported; the error says so explicitly.validateAppendOffset() now accepts a whole-number append offset above the
32-bit integer range instead of coercing it via as.integer() (which produced
NA), and buildTableSelectSQL() emits the offset as a full integer literal
rather than a 32-bit %d. Together these let a resource with more than ~2.1e9
rows stream without the append offset overflowing to NA. Offsets within the
32-bit range are still returned as integer, so index-column narrowing is
unchanged for the common case.transform() test is now skipped on R (< 4.6.0). The package requires
R (>= 4.6.0); on older R the paired S4Vectors ships a transform() whose
internal evaluation-frame stack-walk fails when transform() is called as a
lazily-forced argument promise. This avoids a spurious test ERROR on the
R-oldrel build of an unsupported R version.table() methods for DuckDBTable and DuckDBColumn now declare x as
a formal argument (function(x, ...)), conforming to the table generic in
Bioconductor-devel BiocGenerics, which dispatches on x
(setGeneric("table", signature = "x")) rather than on .... Under the
previous function(...) definition the package failed to install on
Bioc-devel with a conformMethod error ("formal arguments ... omitted in the
method definition cannot be in the signature"). Behavior is unchanged, and the
new signature also conforms against the ...-dispatch generic in the current
Bioconductor release.configureCloud() wires up remote object-storage reads on the shared
connection: it installs + loads the DuckDB httpfs extension up front (so a
firewalled environment fails early with actionable guidance rather than
mid-read) and applies s3_* credential / region settings (s3_region,
s3_access_key_id, s3_secret_access_key, s3_session_token, s3_endpoint,
s3_url_style, s3_use_ssl) resolved from R options (DuckDBDataFrame.s3_*)
or BIOCDUCKDB_S3_* environment variables, applied after httpfs loads
(DuckDB does not know the s3_* settings until then). See ?DuckDBConnection.
A dataset backed by a remote object-storage directory (s3://, gs://,
http(s)://, …) now resolves correctly. Because the VFS cannot be listed with
list.files(), the connection wrapper detects a remote URI, calls
configureCloud(), and lets DuckDB's httpfs glob expand
read_parquet('<uri>/**') — reads only; writing to object storage stays
unsupported (see BiocDuckDB::writeParquet).
configureOutOfCore() sets the DuckDB out-of-core engine knobs
(memory_limit, threads, temp_directory, preserve_insertion_order) on
the shared connection from R options or BIOCDUCKDB_* environment variables
(see ?DuckDBConnection). It runs automatically the first time
acquireDuckDBConn() creates the connection, so a large aggregation / sort /
join can spill instead of being OOM-killed, and a very large export can drop
insertion-order buffering.BIOCDUCKDB_EXTENSION_REPOSITORY (previously
MODL_DUCKDB_EXTENSION_REPOSITORY); the DuckDB-native
DUCKDB_EXTENSION_REPOSITORY fallback is unchanged.writeParquet(..., cluster_by = ) clusters rows on write so DuckDB row-group zonemaps
prune range queries. cluster_by is a character vector (lexicographic ordering),
zorder(cols) (a Morton / Z-order space-filling code over any number of numeric columns,
lowered SQL-side to a generated bit-interleave ORDER BY — no extension needed), or
hilbert(cols) (the native DuckDB spatial ST_Hilbert, exactly two numeric columns,
better locality, requires the spatial extension). Space-filling keys compute per-column
extents with one cheap MIN/MAX scan and bake them into a pushed-down ORDER BY, so the
DuckDB write path never materializes the table. A single physical order clusters one key:
supplying cluster_by overrides the default __index__ ordering. New exported
zorder() / hilbert() constructors and clusterSort() (the in-memory counterpart used
by the materializing data.frame / DataFrame write path). This generalizes the
coord-indexed points layout (DuckDBSpatial) to any multi-dimensional range-queried table
(spatial points, embeddings / reducedDims, genomic-interaction bins).loadExtension(), which installs (if needed) and loads a DuckDB
extension on a connection. It was already used internally to load spatial /
httpfs; exporting it lets companion packages ensure their required extension
is available on the shared connection — e.g. DuckDBSpatial now installs and
loads spatial on load via loadExtension(acquireDuckDBConn(), "spatial").DuckDBTable() and DuckDBDataFrame() gain a collevels argument that
restores factor (and ordered-factor) columns on materialization. Levels are
carried on the object and applied lazily when a column is collected, so
readParquet() can recover factors recorded in a product's schema. Columns
that are cast or otherwise transformed away from a character type are left
untouched.HUGEINT/UHUGEINT) and wide
DECIMAL (precision > 15) collapse to double, and unsigned 64-bit
(UBIGINT) collapses to signed integer64.NA compiled to SQL
NOT IN (..., NULL), which evaluates to UNKNOWN for all rows and returned
an empty result. NA-valued keys are now handled explicitly and never emitted
inside an IN list, reproducing base-R %in% semantics.integer64 keys. Because is.integer()
is FALSE for integer64, the fast-path was previously skipped for exactly
the BIGINT / row-number keycols where it matters most, falling back to an
IN list.SEMI JOIN rather than an
INNER JOIN against the temporary key table. SEMI JOIN is the correct
membership primitive: it neither duplicates rows when the key repeats nor
appends the join column.IN list
to a temporary-table SEMI/ANTI JOIN (from 10000 to 256). Large inline IN
lists dominate DuckDB's SQL compile time, so the crossover belongs in the low
hundreds.INSTALL spatial/httpfs from the public CDN
on every connect (which made an optional, spatial-only concern everyone's
problem). Instead acquireDuckDBConn() now:
BIOCDUCKDB_EXTENSION_REPOSITORY (or the DuckDB-native
DUCKDB_EXTENSION_REPOSITORY);LOADs only extensions already present in the extension directory
(a local, offline-safe operation), so a pre-provisioned or air-gapped cache
just works.loadExtension() is retained as a helper for explicit
or vendored loading.DuckDBTable
abstraction, SQL translation, connection model, and extension points, for
developers).DUCKDB_EXTENSION_DIRECTORY), so extension use works on shared or
read-only R libraries; optional extensions degrade gracefully.BiocCheck: replaced sapply() with
setNames(lapply(...)) where names must be preserved, used seq_len() in
place of 1:n, and tidied condition-signal messages.