Internals & Design
File Interface
The JLDFile object mimics the API of Base.Dict
as much as it can. In particular, keys
, length
, haskey
, isempty
, get
, get!
should work as expected.
#
JLD2.CommittedDatatype
— Type
CommittedDatatype
Reference to a shared datatype message (stored elsewhere in a file). These are stored in the _types
group and indexed.
#
JLD2.CustomSerialization
— Type
CustomSerialization{T,S}
On-disk representation for data that is written as if it were of Julia type T
, but is read as type S
.
#
JLD2.GlobalHeap
— Type
GlobalHeap
Represents an HDF5 global heap structure.
#
JLD2.Group
— Type
Group(file)
JLD2 group object.
Advanced Usage
Takes two optional keyword arguments: estnumentries::Int=4 estlinkname_len::Int=8 These determine how much (additional) empty space should be allocated for the group description. (list of entries) This can be useful for performance when one expects to append many additional datasets after first writing the file.
#
JLD2.Group
— Method
Group(f::JLDFile, name::AbstractString)
Construct an empty group named name
at the top level of JLDFile
f
.
#
JLD2.Group
— Method
Group(g::Group, name::AbstractString)
Construct a group named name
as a child of group g
.
#
JLD2.H5Datatype
— Type
H5Datatype
Supertype of all HDF5 datatypes.
#
JLD2.IndirectPointer
— Type
IndirectPointer
When writing data, we may need to enlarge the memory mapping, which would invalidate any memory addresses arising from the old mmap
pointer. IndirectPointer
holds a pointer to the startptr
field of an MmapIO, and the offset relative to that pointer. It defers computing a memory address until converted to a Ptr{T}, so the memory mapping can be enlarged and addresses will remain valid.
#
JLD2.InlineUnionEl
— Type
InlineUnionEl{T1,T2}(mask::UInt8, t1::T1, t2::T2)
Custom serialization struct for two member isbits union fields e.g. in other structs or arrays. To indicate that t1 is relevant the mask takes the value UInt8(0)
and for t2 UInt8(255)
#
JLD2.JLDFile
— Type
JLDFile{T<:IO}
JLD file object.
#
JLD2.JLDWriteSession
— Type
JLDWriteSession{T}
A JLDWriteSession keeps track of references to serialized objects. If T
is a Dict, h5offset
maps an object ID (returned by calling objectid
) to th RelOffset
of the written dataset. If it is Union{}
, then references are not tracked, and objects referenced multiple times are written multiple times.
#
JLD2.ReadRepresentation
— Type
ReadRepresentation{T,ODR}
A type encoding both the Julia type T
and the on-disk (HDF5) representation ODR
.
#
JLD2.RelOffset
— Type
RelOffset
Represents an HDF5 relative offset. This differs from a file offset (used elsewhere) in that it is relative to the superblock base address. fileoffset
and h5offset
convert between RelOffsets
and file offsets.
SharedDatatype
Reference to a shared datatype message (stored elsewhere in a file).
#
JLD2.TypeMappingException
— Type
constructrr(f::JLDFile, T::DataType, dt::CompoundType, attrs::Vector{ReadAttribute}, hard_failure::Bool=false)
Constructs a ReadRepresentation for a given type. This is the generic method for all types not specially handled below.
If hard_failure is true, then throw a TypeMappingException instead of attempting reconstruction. This helps in cases where we can’t know if reconstructed parametric types will have a matching memory layout without first inspecting the memory layout.
#
JLD2.Upgrade
— Type
Upgrade(T)
Specify an upgrade path for serialized structs using the typemap
.`keyword argument and
rconvert
#
JLD2.behead
— Method
behead(T)
Given a UnionAll type, recursively eliminates the where
clauses
#
JLD2.construct_array
— Method
construct_array{T}(io::IO, ::Type{T}, ::Val{ndims})
Construct array by reading ndims
dimensions from io
. Assumes io
has already been seeked to the correct position.
#
JLD2.fileoffset
— Method
fileoffset(f::JLDFile, x::RelOffset)
Converts an offset x
relative to the superblock of file f
to an absolute offset.
#
JLD2.group_payload_size
— Method
group_payload_size(g)
Returns the size of a group payload, including link info, group info, and link messages, but not the object header. Provides space after the last object message for a continuation message.
#
JLD2.h5offset
— Method
h5offset(f::JLDFile, x::Integer)
Converts an absolute file offset x
to an offset relative to the superblock of file f
.
#
JLD2.jld_finalizer
— Method
jld_finalizer(f::JLDFile)
When a JLDFile is finalized, it is possible that the MmapIO
has been munmapped, since Julia does not guarantee finalizer order. This means that the underlying file may be closed before we get a chance to write to it.
#
JLD2.jldopen
— Function
jldopen(fname::AbstractString, mode::AbstractString; iotype=MmapIO, compress=false, typemap=Dict())
Opens a JLD2 file at path fname
.
"r"
: Open for reading only, failing if no file exists "r+"
: Open for reading and writing, failing if no file exists "w"
/"w+"
: Open for reading and writing, overwriting the file if it already exists "a"
/"a+"
: Open for reading and writing, creating a new file if none exists, but preserving the existing file if one is present
#
JLD2.jldsave
— Method
jldsave(filename, compress=false; kwargs...)
Creates a JLD2 file at filename
and stores the variables given as keyword arguments.
Examples
jldsave("example.jld2"; a=1, b=2, c)
is equivalent to
jldopen("example.jld2, "w") do f f["a"] = 1 f["b"] = 2 f["c"] = c end
To choose the io type IOStream
instead of the default MmapIO
use jldsave(fn, IOStream; kwargs...)
.
#
JLD2.link_size
— Method
link_size(name::String)
Returns the size of a link message, including message header.
#
JLD2.links_size
— Method
links_size(pairs)
Returns the size of several link messages. pairs
is an iterator of String => RelOffset
pairs.
#
JLD2.load_attributes
— Method
load_attributes(f::JLDFile, name::AbstractString)
load_attributes(g::Group, name::AbstractString)
load_attributes(f::JLDFile, offset::RelOffset)
Return a list of attributes attached to the dataset or group.
#
JLD2.load_datatypes
— Method
load_datatypes(f::JLDFile)
Populate f.datatypes and f.jlh5types with all of the committed datatypes from a file. We need to do this before writing to make sure we reuse written datatypes.
#
JLD2.load_object
— Method
load_object(filename)
Returns the only available object from the JLD2 file filename
(The stored object name is inconsequential). If the file contains more than one or no objects, the function throws an ArgumentError
.
Example
To load the only object from the JLD2 file example.jld2:
hello = "world" save_object("example.jld2", hello) hello_loaded = load_object("example.jld2")
#
JLD2.lookup_offset
— Method
lookup_offset(g::Group, name::AbstractString) -> RelOffset
Lookup the offset of a dataset in a group. Returns UNDEFINED_ADDRESS
if the dataset is not present. Does not inspect unwritten_child_groups
.
#
JLD2.pathize
— Method
pathize(g::Group, name::AbstractString, create::Bool) -> Tuple{Group,String}
Converts a path to a group and name object. If create
is true, any intermediate groups will be created, and the dataset name will be checked for uniqueness with existing names.
#
JLD2.prewrite
— Method
prewrite(f::JLDFile)
Check that a JLD file is actually writable, and throw an error if not. Sets the written
flag on the file.
#
JLD2.printtoc
— Method
printtoc([io::IO,] f::JLDFile [; numlines])
Prints an overview of the contents of f
to the IO
.
Use the optional numlines
parameter to restrict the amount of items listed.
#
JLD2.read_array!
— Function
read_array!(v::Array, f::JLDFile, rr)
Fill the array v
with the contents of JLDFile f
at the current position, assuming a ReadRepresentation rr
.
#
JLD2.read_attr_data
— Method
read_attr_data(f::JLDFile, attr::ReadAttribute, expected_datatype::H5Datatype,
rr::ReadRepresentation)
jlread data from an attribute, assuming a specific HDF5 datatype and ReadRepresentation. If the HDF5 datatype does not match, throws an UnsupportedFeatureException
. This allows better type stability while simultaneously validating the data.
#
JLD2.read_attr_data
— Method
read_attr_data(f::JLDFile, attr::ReadAttribute)
jlread data from an attribute.
#
JLD2.read_attribute
— Method
read_attribute(io::IO, f::JLDFile)
Read an attribute message at the current position of the io
object. Supports attribute message version 1 and 2.
#
JLD2.read_compressed_array!
— Function
read_compressed_array!(v::Array, f::JLDFile, rr, data_length::Int, ::Val{filter_id})
Fill the array v
with the compressed contents of JLDFile f
at the current position, assuming a ReadRepresentation rr
and that the compressed data has length data_length
.
#
JLD2.read_data
— Function
read_data(f::JLDFile, dataspace::ReadDataspace, datatype_class::UInt8,
datatype_offset::Int64, data_offset::Int64[, filters::FilterPipeline,
header_offset::RelOffset, attributes::Vector{ReadAttribute}])
Read data from a file. If datatype_class
is typemax(UInt8), the datatype is assumed to be committed, and datatype_offset
points to the offset of the committed datatype’s header. Otherwise, datatype_offset points to the offset of the datatype attribute.
#
JLD2.read_scalar
— Function
read_scalar(f::JLDFile, rr, header_offset::RelOffset)
Read raw data representing a scalar with read representation rr
from the current position of JLDFile f
. header_offset
is the RelOffset of the object header, used to resolve cycles.
#
JLD2.read_size
— Method
read_size(io::IO, flags::UInt8)
Loads a variable-length size according to flags Expects that the first two bits of flags mean: 0 The size of the Length of Link Name field is 1 byte. 1 The size of the Length of Link Name field is 2 bytes. 2 The size of the Length of Link Name field is 4 bytes. 3 The size of the Length of Link Name field is 8 bytes. Returns the size as an Int
#
JLD2.readas
— Method
readas(::Type)::Type
Experimental feature: JLD2.readas
can be overloaded to override which type a saved type is read as, and is used together with custom serialization using JLD2.writeas
.
The typical case is custom serialization of parametric types, where not all type parameters are available during reading. Consider the following example for an anonymous function fun
inside a Foo
struct Foo{F<:Function}
fun::F
end
struct FooSerialization
fun
end
JLD2.writeas(::Type{<:Foo}) = FooSerialization
Base.convert(::Type{<:FooSerialization}, f::Foo) = FooSerialization(f.fun)
JLD2.readas(::Type{<:FooSerialization}) = Foo
struct UndefinedFunction <:Function
fun
end
(f::UndefinedFunction)(args...; kwargs...) = error("The function $(f.fun) is not defined")
function Base.convert(::Type{<:Foo}, f::FooSerialization)
isa(f.fun, Function) && return Foo(f.fun)
return Foo(UndefinedFunction(f.fun))
end
If we include these definitions, call jldsave("foo.jld2"; foo=Foo(x->x^2))
, restart julia, include the definitions again, and call foo = jldopen("foo.jld2") do io; io["foo"]; end
, we get foo::Foo{UndefinedFunction}
and foo::FooSerialization
with and without defining the JLD2.readas
above, respectively.
#
JLD2.save_group
— Method
save_group(g::Group) -> RelOffset
Stores a group to a file, updating it if it has already been saved. Returns UNDEFINED_ADDRESS if the group was already stored, or the offset of the new group otherwise.
#
JLD2.save_object
— Method
save_object(filename, x)
Stores an object x
in a new JLD2 file at filename
. If a file exists at this path, it will be overwritten.
Since the JLD2 format requires that all objects have a name, the object will be stored as single_stored_object
. If you want to store more than one object, use @save
macro, jldopen
or the FileIO API.
Example
To save the string hello
to the JLD2 file example.jld2:
hello = "world" save_object("example.jld2", hello)
#
JLD2.shorttypestring
— Method
shorttypestring(::Type{ <:UnknownType})
Convert an UnknownType to a corresponding string. This is only used to create names for reconstructed types. See also typestring.
#
JLD2.skip_to_aligned!
— Function
skip_to_aligned!(io, rel=0)
Skip to nearest position aligned to a multiple of 8 bytes relative to rel
.
#
JLD2.symbol_length
— Method
symbol_length(x::Symbol)
Returns the length of the string represented by x
.
#
JLD2.typestring
— Method
typestring(::Type{ <:UnknownType})
Convert an UnknownType to a corresponding string. This is only used for warning during reconstruction errors. See also shorttypestring.
#
JLD2.write_link
— Method
write_link(cio, name, offset)
Write a link message at current position in cio
.
#
JLD2.@load
— Macro
@load filename var1 [var2 ...]
Load one or more variables var1,...
from JLD2 file filename
into the current scope and return a vector of the loaded variable names.
For interactive use, the form @load "somefile.jld2"
will load all variables from "somefile.jld2"
into the current scope. This form only supports literal file names and should be avoided in more permanent code so that it’s clear where the variables come from.
Example
To load the variables hello
and foo
from the file example.jld2, use
@load "example.jld2" hello foo
#
JLD2.@save
— Macro
@save filename var1 [var2 ...]
@save filename {compress=true} var1 name2=var2
Write one or more variables var1,...
from the current scope to a JLD2 file filename
.
For interactive use you can save all variables in the current module’s global scope using @save filename
. More permanent code should prefer the explicit form to avoid saving unwanted variables.
Example
To save the string hello
and array xs
to the JLD2 file example.jld2:
hello = "world" xs = [1,2,3] @save "example.jld2" hello xs
For passing options to the saving command use {}
@save "example.jld2" {compress=true} hello xs
For saving variables under a different name use regular assignment syntax
@save "example.jld2" greeting=hello xarray = xs