RCS Header: /cvsroot/petscgraphics/illumulti.c,v 1.25 2003/08/04 13:58:45 hazelsct Exp
This file contains the functions
IlluMultiSave(), IlluMultiLoad() and
IlluMultiRead()
designed to handle distributed storage and retrieval of data on local drives
of machines in a Beowulf cluster. This should allow rapid loading of
timestep data for "playback" from what is essentially a giant RAID-0 array of
distributed disks, at enormously higher speeds than via NFS from a hard drive
or RAID array on the head node. The danger of course is that if one node's
disk goes down, you don't have a valid data set any more, but that's the
nature of RAID-0, right?
The filenames saved are:
- <basename>.cpu####.meta, where ####
is replaced by the CPU number (more than four digits if there are more than
9999 CPUs :-), which has the metadata for the whole thing in XML format
(written by
GNOME libxml),
as described in the notes on the
IlluMultiStoreXML() function.
- <basename>.cpu####.data
which is simply a stream of the raw data, optionally compressed by
gzip.
If one were saving timesteps, one might include a timestep number in the
basename, and also timestep and simulation time in the metadata. The
metadata can also hold simulation parameters, etc.
This supports 1-D, 2-D and 3-D distributed arrays. As an extra feature, you
can load a multi-CPU distributed array scattered over lots of files into a
single CPU, to facilitate certain modes of data visualization.
Included Files
- #include </usr/lib/petsc/include/petscblaslapack.h>
Preprocessor definitions
#define DPRINTF( fmt, args... )
#define __FUNCT__ "IlluMultiParseXML"
#define __FUNCT__ "IlluMultiParseData"
#define __FUNCT__ "IlluMultiStoreXML"
#define __FUNCT__ "IlluMultiStoreData"
#define __FUNCT__ "checkagree"
#define __FUNCT__ "IlluMultiRead"
#define __FUNCT__ "IlluMultiLoad"
#define __FUNCT__ "IlluMultiSave"
This creates a new distributed array of the appropriate size and loads the
data into the vector contained in it (as retrieved by
DAGetVector()).
It also reads the user metadata parameters into arrays stored at the supplied
pointers.
int IlluMultiLoad ( char* basename, DA* theda, PetscScalar* wx, PetscScalar* wy, PetscScalar* wz, field_plot_type** fieldtypes, int* usermetacount, char*** usermetanames, char*** usermetadata )
- int IlluMultiLoad
- It returns zero or an error code.
- char* basename
- Base file name.
- DA* theda
- Pointer to a DA object (to be created by this function).
- PetscScalar* wx
- Physical overall width in the
x-direction.
- PetscScalar* wy
- Physical overall width in the
y-direction.
- PetscScalar* wz
- Physical overall width in the
z-direction.
- field_plot_type** fieldtypes
- Data (plot) types for field variables.
- int* usermetacount
- Pointer to an int where we put the number of user metadata
parameters loaded.
- char*** usermetanames
- Pointer to a char ** where the loaded parameter names
are stored. This is
malloced by this function, so a call to free()
is needed to free up its data.
- char*** usermetadata
- Pointer to a char ** where the loaded parameter strings
are stored. This is
malloced by this function, so a call to free()
is needed to free up its data.
First it gets the parameters from the XML file.
Next it creates a distributed array based on those parameters, and sets
the names of its fields.
Then it streams the data into the distributed array's vector in one big
slurp.
This reads the data into an existing distributed array and vector, checking
that the sizes are right etc.
int IlluMultiRead ( DA theda, Vec X, char* basename, int* usermetacount, char*** usermetanames, char*** usermetadata )
- int IlluMultiRead
- It returns zero or an error code.
- DA theda
- Distributed array object controlling the data to read.
- Vec X
- Vector into which to read the data.
- char* basename
- Base file name.
- int* usermetacount
- Pointer to an int where we put the number of user metadata
parameters loaded.
- char*** usermetanames
- Pointer to a char ** where the loaded parameter names
are stored. This is
malloced by this function, so a call to free()
is needed to free up its data.
- char*** usermetadata
- Pointer to a char ** where the loaded parameter strings
are stored. This is
malloced by this function, so a call to free()
is needed to free up its data.
First it gets the properties of the distributed array for comparison with
the metadata.
Next it parses the XML metadata file into the document tree, and reads
its content into the appropriate structures, comparing parameters with
those of the existing distributed array structure.
Then it streams in the data in one big slurp.
This saves the vector X in multiple files, two per process. Note the use of
PETSC_COMM_WORLD,
that will have to be corrected if somebody wants a different communicator
(maybe get it from the DA?).
int IlluMultiSave ( DA theda, Vec X, char* basename, PetscScalar wx, PetscScalar wy, PetscScalar wz, field_plot_type* fieldtypes, int usermetacount, char** usermetanames, char** usermetadata, int compressed )
- int IlluMultiSave
- it returns zero or an error code.
- DA theda
- Distributed array object controlling data saved.
- Vec X
- Vector whose data are actually being saved.
- char* basename
- Base file name.
- PetscScalar wx
-
- PetscScalar wy
-
- PetscScalar wz
-
- field_plot_type* fieldtypes
-
- int usermetacount
- Number of user metadata parameters.
- char** usermetanames
- User metadata parameter names.
- char** usermetadata
- User metadata parameter strings.
- int compressed
- Data compression: if zero then no compression (fastest), 1-9
then gzip compression level, 10-15 then gzip --best. If 16-31 then save
guint32s representing relative values between min and max for each field,
compressed according to this value minus 16. Likewise for 32-47 and
guint16s, and 48-63 for guint8s. Yes, these alternative formats lose
information and can't be used for accurate checkpointing, but they should
retain enough data for visualization (except perhaps for the guint8s, which
are possibly acceptable for vectors but certainly not contours).
First a check to verify a supported value of
compressed,
but no fancy guint* compression for complex!
Then get the distributed array parameters and processor number, and store
all this data in the XML .meta file.
Finally, the data just stream out to the data file or gzip pipe in one
big lump.
This function reads in the data stored by IlluMultiStoreData(), complete with
int/gzip compression.
static int IlluMultiParseData ( PetscScalar* globalarray, char* basename, int rank, int compressed, int gridpoints, int dof, int wrongendian, PetscScalar* fieldmin, PetscScalar* fieldmax )
- int IlluMultiParseData
- It returns zero or an error code.
- PetscScalar* globalarray
- Array into which to load the (local) data.
- char* basename
- Base file name.
- int rank
- CPU number to read data for.
- int compressed
- Data compression: if zero then no compression (fastest), 1-9
then gzip compression level, 10-15 then gzip --best. If 16-31 then save
guint32s representing relative values between min and max for each field,
compressed according to this value minus 16. Likewise for 32-47 and
guint16s, and 48-63 for guint8s. Yes, these alternative formats lose
information and can't be used for accurate checkpointing, but they should
retain enough data for visualization (except perhaps for the guint8s, which
are possibly acceptable for vectors but likely not contours).
- int gridpoints
- Number of gridpoints to read data for.
- int dof
- Degrees of freedom at each node, a.k.a. number of field variables.
- int wrongendian
- Tells whether the data are stored in the opposite endian
format from this platform, and thus must be switched when the data are
streamed in.
- PetscScalar* fieldmin
- Minimum value of each field variable.
- PetscScalar* fieldmax
- Maximum value of each field variable.
This function reads in the XML metadata document and returns the various
parameter values in the addresses pointed to by the arguments. It is called
by IlluMultiLoad() and IlluMultiRead().
static int IlluMultiParseXML ( char* basename, int rank, int* compressed, int* wrongendian, int* dim, int* px, int* py, int* pz, int* nx, int* ny, int* nz, PetscScalar* wx, PetscScalar* wy, PetscScalar* wz, int* xm, int* ym, int* zm, int* dof, int* sw, DAStencilType* st, DAPeriodicType* wrap, char*** fieldnames, field_plot_type** fieldtypes, PetscScalar** fieldmin, PetscScalar** fieldmax, int* usermetacount, char*** usermetanames, char*** usermetadata )
- int IlluMultiParseXML
- It returns zero or an error code.
- char* basename
- Base file name.
- int rank
- CPU number to read data for.
- int* compressed
- Data compression: if zero then no compression (fastest), 1-9
then gzip compression level, 10-15 then gzip --best. If 16-31 then save
guint32s representing relative values between min and max for each field,
compressed according to this value minus 16. Likewise for 32-47 and
guint16s, and 48-63 for guint8s. Yes, these alternative formats lose
information and can't be used for accurate checkpointing, but they should
retain enough data for visualization (except perhaps for the guint8s, which
are possibly acceptable for vectors but likely not contours).
- int* wrongendian
- Tells whether the data are stored in the opposite endian
format from this platform, and thus must be switched when the data are
streamed in.
- int* dim
- Dimensionality of the space.
- int* px
- Number of processors in the
x-direction.
- int* py
- Number of processors in the
y-direction.
- int* pz
- Number of processors in the
z-direction.
- int* nx
- Number of grid points over the entire array in the
x-direction.
- int* ny
- Number of grid points over the entire array in the
y-direction.
- int* nz
- Number of grid points over the entire array in the
z-direction.
- PetscScalar* wx
- Physical overall width in the
x-direction, PETSC_NULL if not needed.
- PetscScalar* wy
- Physical overall width in the
y-direction, PETSC_NULL if not needed.
- PetscScalar* wz
- Physical overall width in the
z-direction, PETSC_NULL if not needed.
- int* xm
- Number of grid points over the local part of the array in the
x-direction.
- int* ym
- Number of grid points over the local part of the array in the
y-direction.
- int* zm
- Number of grid points over the local part of the array in the
z-direction.
- int* dof
- Degrees of freedom at each node, a.k.a. number of field variables.
- int* sw
- Stencil width.
- DAStencilType* st
- Stencil type, given by the
PETSc enum values.
- DAPeriodicType* wrap
- Periodic type, given by the
PETSc enum values.
- char*** fieldnames
- Names of the field variables.
- field_plot_type** fieldtypes
- Data (plot) types for field variables,
PETSC_NULL if not needed.
- PetscScalar** fieldmin
- Minimum value of each field variable.
- PetscScalar** fieldmax
- Maximum value of each field variable.
- int* usermetacount
- Number of user metadata parameters.
- char*** usermetanames
- User metadata parameter names.
- char*** usermetadata
- User metadata parameter strings.
For GlobalSize, since there's no *size attribute (for an 0.1
version document), assume 1.
If the type attribute is missing from the Field node (as it is in
version 0.1 documents), assume
FIELD_SCALAR_COLORS.
This function stores the data file.
static int IlluMultiStoreData ( PetscScalar* globalarray, char* basename, int rank, int compressed, int gridpoints, int dof, PetscScalar* fieldmin, PetscScalar* fieldmax )
- int IlluMultiStoreData
- It returns zero or an error code.
- PetscScalar* globalarray
- Array from which to save the (local) data.
- char* basename
- Base file name.
- int rank
- CPU number to read data for.
- int compressed
- Data compression: if zero then no compression (fastest), 1-9
then gzip compression level, 10-15 then gzip --best. If 16-31 then save
guint32s representing relative values between min and max for each field,
compressed according to this value minus 16. Likewise for 32-47 and
guint16s, and 48-63 for guint8s. Yes, these alternative formats lose
information and can't be used for accurate checkpointing, but they should
retain enough data for visualization (except perhaps for the guint8s, which
are possibly acceptable for vectors but likely not contours).
- int gridpoints
- Number of gridpoints to store data for.
- int dof
- Degrees of freedom at each node, a.k.a. number of field variables.
- PetscScalar* fieldmin
- Minimum value of each field variable.
- PetscScalar* fieldmax
- Maximum value of each field variable.
This function opens, stores and closes the XML metadata file for IlluMulti
format storage. It is called by IlluMultiSave().
static int IlluMultiStoreXML ( char* basename, int rank, int compressed, int dim, int px, int py, int pz, int nx, int ny, int nz, PetscScalar wx, PetscScalar wy, PetscScalar wz, int xm, int ym, int zm, int dof, int sw, int st, int wrap, char** fieldnames, field_plot_type* fieldtypes, PetscReal* fieldmin, PetscReal* fieldmax, int usermetacount, char** usermetanames, char** usermetadata )
- int IlluMultiStoreXML
- It returns zero or an error code.
- char* basename
- Base file name.
- int rank
- CPU number to store data for.
- int compressed
- Data compression: if zero then no compression (fastest), 1-9
then gzip compression level, 10-15 then gzip --best. If 16-31 then save
guint32s representing relative values between min and max for each field,
compressed according to this value minus 16. Likewise for 32-47 and
guint16s, and 48-63 for guint8s. Yes, these alternative formats lose
information and can't be used for accurate checkpointing, but they should
retain enough data for visualization (except perhaps for the guint8s, which
are possibly acceptable for vectors but certainly not contours).
- int dim
- Dimensionality of the space.
- int px
- Number of processors in the
x-direction.
- int py
- Number of processors in the
y-direction.
- int pz
- Number of processors in the
z-direction.
- int nx
- Number of grid points over the entire array in the
x-direction.
- int ny
- Number of grid points over the entire array in the
y-direction.
- int nz
- Number of grid points over the entire array in the
z-direction.
- PetscScalar wx
- Physical overall width in the
x-direction.
- PetscScalar wy
- Physical overall width in the
y-direction.
- PetscScalar wz
- Physical overall width in the
z-direction.
- int xm
- Number of grid points over the local part of the array in the
x-direction.
- int ym
- Number of grid points over the local part of the array in the
y-direction.
- int zm
- Number of grid points over the local part of the array in the
z-direction.
- int dof
- Degrees of freedom at each node, a.k.a. number of field variables.
- int sw
- Stencil width.
- int st
- Stencil type, given by the
PETSc enum values.
- int wrap
- Periodic type, given by the
PETSc enum values.
- char** fieldnames
- Names of the field variables.
- field_plot_type* fieldtypes
- Data (plot) types for field variables.
- PetscReal* fieldmin
- Minimum value of each field variable.
- PetscReal* fieldmax
- Maximum value of each field variable.
- int usermetacount
- Number of user metadata parameters.
- char** usermetanames
- User metadata parameter names.
- char** usermetadata
- User metadata parameter strings.
The XML tags in the .meta file consist of:
IlluMulti | Primary tag, with attributes
version, endian (big or
little) and compression (none, 1-9, best;
long*, short* or char*) |
GlobalCPUs | Number of CPUs in each
direction, with attributes dimensions, xwidth,
ywidth and zwidth |
GlobalSize | Size of the entire distributed
array, with attributes xwidth, ywidth,
zwidth, xsize**, ysize**,
zsize** and fields |
LocalSize | Size of the local part of the
array, with attributes xwidth, ywidth and
zwidth |
Stencil | Stencil and periodic data, with
attributes width, type and periodic
(using PETSc enum values) |
Field | Data on each field, attributes
name, type*, min and
max |
User | User parameters, attributes
name and value |
*Lossy compression to smaller data types.
**Represents new attribute for IlluMulti 0.2 file format.
Ancillary routine for
IlluMultiRead():
checks agreement of parameters and reports disagreement if necessary.
static inline int checkagree ( int da, int file, char* parameter )
- int checkagree
- Returns 0 if they agree, 1 otherwise.
- int da
- Integer parameter from the existing DA.
- int file
- Integer parameter read from the file.
- char* parameter
- Parameter name for reporting.