API

The VISSSlib processing library provides tools for analyzing Video In Situ Snowfall Sensor (VISSS) data. The library is organized into several modules, each serving a specific purpose in the data processing pipeline.

class VISSSlib.analysis.matchGUI(case, config, markParticles=True, increaseContrast=False, showTracks=False, skipNonMatched=False, showVars=['Dmax'], lv1match=None, showParticles=False, scale=0.5)

Bases: object

GUI for stereo view matching visualization.

Provides interactive GUI controls for navigating and visualizing stereo camera matching data.

Attributes

sv_stereoViewMatch

Stereo view matcher instance

showVarslist

Variables to display in the GUI

showParticlesbool

Whether to show particle details

scalefloat

Scaling factor for displayed images

updateHandlesId(fid)

Update GUI handles for a specific frame ID.

Parameters

fidint

Frame ID to update

Returns

tuple

Updated frame, meta frames, level 1 detects, level 1 matches, particles

updateHandles(frame, metaFrames, lv1detects, lv1matches, particles)

Update GUI handles with new data.

Parameters

framearray

Frame data to display

metaFrameslist

Meta frames data

lv1detectslist

Level 1 detect data

lv1matcheslist

Level 1 match data

particlesdict

Particle data

getNN()

Get current frame number from GUI input.

Returns

int

Current frame number

setNN(nn)

Set current frame number in GUI.

Parameters

nnint

Frame number to set

createGUI(pid=0, startId=0)

Create the GUI interface.

Parameters

pidint, optional

Starting PID (default: 0)

startIdint, optional

Starting ID (default: 0)

class VISSSlib.analysis.manualMatchGUI(case, config, markParticles=True)

Bases: object

Manual matching GUI for stereo view analysis.

Provides interactive GUI controls for manual stereo camera matching.

Attributes

sv_stereoViewDetect

Stereo view detector instance

updateHandles(frame, metaFrames, lv1detects)

Update GUI handles with new data.

Parameters

framearray

Frame data to display

metaFrameslist

Meta frames data

lv1detectslist

Level 1 detect data

getNN()

Get current frame numbers from GUI inputs.

Returns

dict

Dictionary mapping camera names to frame numbers

setNN(nn)

Set current frame numbers in GUI.

Parameters

nndict

Dictionary mapping camera names to frame numbers

createGUI()

Create the manual matching GUI interface.

class VISSSlib.av.VideoReaderMeta(movFilePattern, metaFrames, lv1detect=None, lv1match=None, imagesL1detect=None, safeMode=False, config=None, skipNonMatched=False)

Bases: object

Metadata manager for video readers with associated data.

This class manages video reading operations along with associated metadata and detection data for stereo camera systems. It handles multiple threads and provides methods for frame retrieval with particle annotations.

Attributes

metaFramesxarray.Dataset

Metadata about frames including capture times and thread information

lv1detectxarray.Dataset or None

Level 1 detection data

lv1matchxarray.Dataset or None

Level 1 matching data

tarFileobject or None

Archive file handle for particle images

safeModebool

Whether to enable safe mode for frame seeking

configobject or None

Configuration object

skipNonMatchedbool

Whether to skip non-matched particles

movFilePatternstr

Pattern for video file names

threadsarray

Unique thread identifiers

videodict

Dictionary mapping thread IDs to video readers

positionint

Current frame position

positionsdict

Dictionary mapping thread IDs to their current positions

currentThreadint or None

Currently active thread ID

currentFramearray or None

Current frame data

currentMetaFramesxarray.Dataset or None

Current metadata frames

currentlv1detectxarray.Dataset or None

Current level 1 detection data

currentPidsarray or None

Current particle IDs

resetVideo()

Reset video readers by releasing and reopening them.

This method releases all current video readers and reopens them, useful when video files need to be reloaded or reset.

getNextFrame(markParticles=False)

Get the next frame from the video sequence.

Parameters

markParticlesbool, optional

Whether to mark particles on the frame, default is False

Returns

tuple

Frame data depending on markParticles parameter

getPrevFrame(markParticles=False)

Get the previous frame from the video sequence.

Parameters

markParticlesbool, optional

Whether to mark particles on the frame, default is False

Returns

tuple

Frame data depending on markParticles parameter

getFrameByIndex(ii, increaseContrast=False)

Retrieve frame by index with associated metadata.

Parameters

iiint

Frame index to retrieve

increaseContrastbool, optional

Whether to increase image contrast, default is False

Returns

tuple

A tuple containing (success_flag, frame_data, metadata) where success_flag indicates if the operation was successful, frame_data is the retrieved frame or None, and metadata is the associated metadata or None.

getFrameByCaptureTime(captureTime, increaseContrast=False)

Retrieve frame by capture time with associated metadata.

Parameters

captureTimedatetime64

Capture time to retrieve frame for

increaseContrastbool, optional

Whether to increase image contrast, default is False

Returns

tuple

A tuple containing (success_flag, frame_data, metadata) where success_flag indicates if the operation was successful, frame_data is the retrieved frame or None, and metadata is the associated metadata or None.

getFrameByCaptureTimeWithParticles(captureTime, pad=4, markParticles=False, highlightPid=None, increaseContrast=False, showTracks=False)

Retrieve frame by capture time with particle annotations.

Parameters

captureTimedatetime64

Capture time to retrieve frame for

padint, optional

Padding around particle bounding boxes, default is 4

markParticlesbool, optional

Whether to mark particles on the frame, default is False

highlightPidint or str, optional

Particle ID to highlight, default is None

increaseContrastbool, optional

Whether to increase image contrast, default is False

showTracksbool, optional

Whether to show particle tracks, default is False

Returns

tuple

A tuple containing (success_flag, annotated_frame, metadata, detection_data, matched_data, particles) where: - success_flag indicates if the operation was successful - annotated_frame is the frame with annotations or None - metadata is the associated metadata or None - detection_data is the detection data or None - matched_data is the matched data or None - particles is a dictionary of particle images or None

getFrameByIndexWithParticles(ii, markParticles=False, highlightPid=None, increaseContrast=False)

Retrieve frame by index with particle annotations.

Parameters

iiint

Frame index to retrieve

markParticlesbool, optional

Whether to mark particles on the frame, default is False

highlightPidint or str, optional

Particle ID to highlight, default is None

increaseContrastbool, optional

Whether to increase image contrast, default is False

Returns

tuple

A tuple containing (success_flag, annotated_frame, metadata, detection_data, matched_data, particles) where: - success_flag indicates if the operation was successful - annotated_frame is the frame with annotations or None - metadata is the associated metadata or None - detection_data is the detection data or None - matched_data is the matched data or None - particles is a dictionary of particle images or None

property currentCaptureTime

Get the current capture time.

Returns

datetime64 or None

Current capture time or None if no metadata is available.

property total_frames

Get the total number of frames across all videos.

Returns

int

Total number of frames in all video streams.

release()

Release all video resources.

This method releases all video readers and closes archive files to free up system resources.

getParticle(pid, heightOffset=64)

Retrieve a specific particle image.

Parameters

pidint

Particle ID to retrieve

heightOffsetint, optional

Height offset for particle cropping, default is 64

Returns

tuple

A tuple containing (cropped_particle_image, full_frame_image)

VISSSlib.detection.logDebug = True

changes in 11/23

blurSigma=1 isntead 1.5, #makes particle a bit sharper, no disadvantage found so far minArea=0 instead 1, #changed!! minDmax=0 instead 2, #changed!! doubleDynamicRange=True, # does improve detection, problematic for wings joinCannyEdges=False, # turned out to be not working check4childCntLength = True changed! avoids erode filter of fgmask bug fixes regarding handling of cnt children

class VISSSlib.detection.detectedParticles(config, pidOffset=0, verbosity=0, testing=[])

Bases: object

Class for detecting and managing particles in video sequences.

This class handles the detection of particles in video frames using background subtraction and contour analysis. It manages particle properties and maintains state between frames.

Attributes

configobject

Configuration object containing detection parameters

verbosityint

Verbosity level for logging

testinglist

Testing modes to enable during processing

maxNParticleint

Maximum number of particles allowed per frame

alldict

Dictionary storing all detected particles by frame index

lastFramedict

Dictionary storing particles from the last processed frame

ppint

Particle ID counter

fgMaskarray

Foreground mask from background subtraction

nMovingPixint

Number of moving pixels in current frame

nMovingPix2int

Number of moving pixels after processing

blurint

Blur estimation for current frame

capture_idint

Capture ID for current frame

record_idint

Record ID for current frame

capture_timedatetime64

Capture time for current frame

record_timedatetime64

Record time for current frame

nThreadint

Thread identifier for current frame

nParticleint

Number of particles detected in current frame

backSubobject

Background subtractor instance

update(frame, pp, capture_id, record_id, capture_time, record_time, nThread, training=False, blockingThresh=None)

Update particle detection for a new frame.

Parameters

framearray

Input frame to process

ppint

Frame index

capture_idint

Capture ID for the frame

record_idint

Record ID for the frame

capture_timedatetime64

Capture time for the frame

record_timedatetime64

Record time for the frame

nThreadint

Thread identifier for the frame

trainingbool, optional

Whether to run in training mode, default is False

blockingThreshint, optional

Threshold for blocking frames, default is None

Returns

tuple

A tuple containing (success_flag, number_of_particles) where success_flag indicates if processing was successful and number_of_particles is the count of detected particles or error code

applyCannyFilter(frame, fgMask, threshold1=0, threshold2=25)

Apply Canny edge detection filter to enhance particle detection.

Parameters

framearray

Input frame to process

fgMaskarray

Foreground mask

threshold1int, optional

Lower threshold for Canny edge detection, default is 0

threshold2int, optional

Upper threshold for Canny edge detection, default is 25

Returns

array

Filtered foreground mask

add(frame1, fgMask, cnt, **kwargs)

Add a new particle to the detection results.

Parameters

frame1array

Input frame

fgMaskarray

Foreground mask

cntarray

Contour coordinates

**kwargsdict

Additional keyword arguments

Returns

bool

True if particle was successfully added, False otherwise

collectResults(includeCnts=False)

Collect and organize detection results into an xarray Dataset.

Parameters

includeCntsbool, optional

Whether to include contour data, default is False

Returns

xarray.Dataset

Dataset containing all particle properties

property N

Get the number of detected particles.

Returns

int

Number of detected particles

property pids

Get the list of particle IDs.

Returns

list

List of particle IDs

class VISSSlib.detection.singleParticle(parent, config, capture_id, record_id, capture_time, record_time, nThread, pp1, frame1, mask1, cnt, cntChild, xOffset, yOffset, testing, verbosity=0)

Bases: object

Class representing a single detected particle.

This class stores all properties and characteristics of a detected particle including geometric features, intensity statistics, and contour information.

Attributes

pidint

Particle ID

record_idint

Record ID for the frame

capture_idint

Capture ID for the frame

capture_timedatetime64

Capture time for the frame

record_timedatetime64

Record time for the frame

nThreadint

Thread identifier for the frame

cntarray

Contour coordinates

cntChildlist

Child contour coordinates

versionstr

Version string

testinglist

Testing modes enabled

xOffsetint

X offset for ROI

yOffsetint

Y offset for ROI

roiarray

Region of interest coordinates

particleBoxMaskarray

Particle mask

particleBoxarray

Particle image data

particleBoxAlphaarray

Particle image with alpha channel

particleBoxCroppedarray

Cropped particle image

pixMinfloat

Minimum pixel value

pixMaxfloat

Maximum pixel value

pixMeanfloat

Mean pixel value

pixPercentilesarray

Pixel value percentiles

pixStdfloat

Standard deviation of pixel values

pixSkewfloat

Skewness of pixel values

pixKurtosisfloat

Kurtosis of pixel values

particleContrastfloat

Contrast of particle relative to background

perimeterErodedfloat

Perimeter after erosion

position_upperLefttuple

Upper left position of bounding box

Droituple

Dimensions of bounding box

position_circletuple

Center position of minimum enclosing circle

Dmaxfloat

Diameter of minimum enclosing circle

pixCenterint

Pixel value at center

position_recttuple

Position of minimum area rectangle

Dfit_recttuple

Dimensions of minimum area rectangle

angle_rectfloat

Angle of minimum area rectangle

ellipseDirecttuple

Direct ellipse fit parameters

ellipsetuple

Ellipse fit parameters

position_ellipsetuple

Position of fitted ellipse

position_ellipseDirecttuple

Position of direct ellipse fit

Dfit_ellipsetuple

Dimensions of fitted ellipse

Dfit_ellipseDirecttuple

Dimensions of direct ellipse fit

angle_ellipsefloat

Angle of fitted ellipse

angle_ellipseDirectfloat

Angle of direct ellipse fit

position_fitlist

Positions of different fits

Dfitlist

Dimensions of different fits

anglelist

Angles of different fits

aspectRatiotuple

Aspect ratios from different fits

areafloat

Area of particle

perimeterfloat

Perimeter of particle

areaConsideringHolesfloat

Area considering internal holes

perimeterConsideringHolesfloat

Perimeter considering internal holes

position_centroidtuple

Centroid position

blurfloat

Blur estimation

contourFFTsumfloat

Sum of FFT coefficients

contourFFTstdfloat

Standard deviation of FFT coefficients

contourFFTarray

FFT coefficients

FFTfreqsarray

FFT frequencies

solidityfloat

Solidity measure

solidityConsideringHolesfloat

Solidity considering holes

extentfloat

Extent measure

extentConsideringHolesfloat

Extent considering holes

successbool

Whether particle detection was successful

dropImages()

Free memory by dropping image references.

drawContour(frame)

Draw particle contours on a frame.

Parameters

framearray

Input frame to draw contours on

Returns

array

Frame with contours drawn

annotate(frame, color=(0, 255, 0), extra='')

Annotate a frame with particle information.

Parameters

framearray

Input frame to annotate

colortuple, optional

Annotation color, default is (0, 255, 0)

extrastr, optional

Extra annotation text, default is “”

Returns

array

Annotated frame

getAnnotatedParticle(frame, extra=20)

Get an annotated particle image.

Parameters

framearray

Input frame

extraint, optional

Extra padding around particle, default is 20

Returns

array

Annotated particle image

VISSSlib.detection.extractRoi(roi, frame, extra=0)

Extract region of interest from frame.

Parameters

roituple

Region of interest coordinates (x, y, w, h)

framearray

Input frame

extraint, optional

Extra padding, default is 0

Returns

tuple

Extracted ROI, x offset, y offset, new ROI

VISSSlib.detection.checkMotion(subFrame, oldFrame, threshs)

Check whether something is moving - identical to VISSS C code.

Parameters

subFramearray

Subtracted frame

oldFramearray

Previous frame

threshsarray

Threshold values for motion detection

Returns

array

Number of changed pixels for each threshold

VISSSlib.detection.detectParticles(fname, config, testing=[], writeNc=True, stopAfter=None, version='0.0', skipExisting=True, verbosity=0)

Detect particles in a video file.

Parameters

fnamestr

Filename of the video file to process

configdict

VISSS configuration dictionary

testinglist, optional

Testing modes to enable, default is []

writeNcbool, optional

Whether to write NetCDF output, default is True

stopAftertuple or int, optional

Stop processing after specified frame, default is None

versionstr, optional

Version string, default is __version__

skipExistingbool, optional

Whether to skip existing files, default is True

verbosityint, optional

Verbosity level, default is 0

Returns

xarray.Dataset or None

Detected particle data or None if no data

VISSSlib.detection.joinEdges(mask)

Join edges in a binary mask to close gaps.

Parameters

maskarray

Binary mask to process

Returns

array

Processed mask with joined edges

VISSSlib.detection.splitUpConours(cntsTmp, hierarchy)

Reorder contours based on hierarchy.

Parameters

cntsTmplist

List of contour arrays

hierarchyarray

Contour hierarchy information

Returns

tuple

Tuple containing (cnts, cntChildren) where cnts are the reordered contours and cntChildren are the child contours grouped by parent

VISSSlib.distributions.createLevel2detect(case, camera, config, DbinsPixel=range(0, 301), sizeDefinitions=['Dmax', 'Dequiv'], endTime=np.timedelta64(1, 'D'), skipExisting=True, writeNc=True, applyFilters=[], doPlot=True, doParticlePlot=True)

Create level2detect data for a given case and camera.

Processes raw particle detection data to generate statistical distributions and quality metrics for precipitation analysis.

Parameters

casestr

Case identifier. This can be a date string (YYYYMMDD) or date range (YYYYMMDD-YYYYMMDD) or comma-separated dates (YYYYMMDD,YYYYMMDD,YYYYMMDD).

camerastr

Camera identifier. this can be also “all”, “leader”, or “follower”.

configdict

Configuration settings.

DbinsPixelrange, optional

Pixel bins for size distribution, defaults to range(301).

sizeDefinitionslist, optional

Size definitions, defaults to [“Dmax”, “Dequiv”].

endTimenumpy.timedelta64, optional

End time for processing, defaults to np.timedelta64(1, “D”).

skipExistingbool, optional

Skip existing outputs, defaults to True.

writeNcbool, optional

Write NetCDF outputs, defaults to True.

applyFilterslist, optional

Filters to apply, defaults to []. Each filter is a tuple with: 1) variable name (e.g. aspectRatio, all lv1 variables work) 2) Operator, one of ‘>’,’<’,’>=’,’<=’,’==’ 3) Value for comparison or a list of two values, used as intercept and slope (in this order) for comparison with a linear function of Dmax 4) if variable contains extra dimensions, which one to select, {} otherwise

Example to get all particles > 10 pixels (using max of both cameras) with aspectRatio >= 0.7 (using min of both cameras) applyFilters = [

(“Dmax”,”>”,10,”max”,{}), (“aspectRatio”,”>”,0.7,”min”,{“fitMethod”:’cv2.fitEllipseDirect’}),

]

doPlotbool, optional

Create plots, defaults to True.

doParticlePlotbool, optional

Create particle plots, defaults to True.

Returns

tuple

Processed dataset and file path.

VISSSlib.distributions.createLevel2match(case, config, DbinsPixel=range(0, 301), sizeDefinitions=['Dmax', 'Dequiv'], endTime=np.timedelta64(1, 'D'), skipExisting=True, writeNc=True, applyFilters=[], doPlot=True, doParticlePlot=True)

Create level2match data for a given case and camera.

Processes raw particle detection data to generate statistical distributions and quality metrics for precipitation analysis.

Parameters

casestr

Case identifier. This can be a date string (YYYYMMDD) or date range (YYYYMMDD-YYYYMMDD) or comma-separated dates (YYYYMMDD,YYYYMMDD,YYYYMMDD).

configdict

Configuration settings.

DbinsPixelrange, optional

Pixel bins for size distribution, defaults to range(301).

sizeDefinitionslist, optional

Size definitions, defaults to [“Dmax”, “Dequiv”].

endTimenumpy.timedelta64, optional

End time for processing, defaults to np.timedelta64(1, “D”).

skipExistingbool, optional

Skip existing outputs, defaults to True.

writeNcbool, optional

Write NetCDF outputs, defaults to True.

applyFilterslist, optional

Filters to apply, defaults to []. Each filter is a tuple with: 1) variable name (e.g. aspectRatio, all lv1 variables work) 2) Operator, one of ‘>’,’<’,’>=’,’<=’,’==’ 3) Value for comparison or a list of two values, used as intercept and slope (in this order) for comparison with a linear function of Dmax 4) if variable contains extra dimensions, which one to select, {} otherwise

Example to get all particles > 10 pixels (using max of both cameras) with aspectRatio >= 0.7 (using min of both cameras) applyFilters = [

(“Dmax”,”>”,10,”max”,{}), (“aspectRatio”,”>”,0.7,”min”,{“fitMethod”:’cv2.fitEllipseDirect’}),

]

doPlotbool, optional

Create plots, defaults to True.

doParticlePlotbool, optional

Create particle plots, defaults to True.

Returns

tuple

Processed dataset and file path.

VISSSlib.distributions.createLevel2track(case, config, DbinsPixel=range(0, 301), sizeDefinitions=['Dmax', 'Dequiv'], endTime=np.timedelta64(1, 'D'), skipExisting=True, writeNc=True, applyFilters=[], doPlot=True)

Create level2track data for a given case and camera.

Processes raw particle detection data to generate statistical distributions and quality metrics for precipitation analysis.

Parameters

casestr

Case identifier. This can be a date string (YYYYMMDD) or date range (YYYYMMDD-YYYYMMDD) or comma-separated dates (YYYYMMDD,YYYYMMDD,YYYYMMDD).

configdict

Configuration settings.

DbinsPixelrange, optional

Pixel bins for size distribution, defaults to range(301).

sizeDefinitionslist, optional

Size definitions, defaults to [“Dmax”, “Dequiv”].

endTimenumpy.timedelta64, optional

End time for processing, defaults to np.timedelta64(1, “D”).

skipExistingbool, optional

Skip existing outputs, defaults to True.

writeNcbool, optional

Write NetCDF outputs, defaults to True.

applyFilterslist, optional

Filters to apply, defaults to []. Each filter is a tuple with: 1) variable name (e.g. aspectRatio, all lv1 variables work) 2) Operator, one of ‘>’,’<’,’>=’,’<=’,’==’ 3) Value for comparison or a list of two values, used as intercept and slope (in this order) for comparison with a linear function of Dmax 4) if variable contains extra dimensions, which one to select, {} otherwise

Example to get all particles > 10 pixels (using max of both cameras) with aspectRatio >= 0.7 (using min of both cameras) applyFilters = [

(“Dmax”,”>”,10,”max”,{}), (“aspectRatio”,”>”,0.7,”min”,{“fitMethod”:’cv2.fitEllipseDirect’}),

]

doPlotbool, optional

Create plots, defaults to True.

doParticlePlotbool, optional

Create particle plots, defaults to True.

Returns

tuple

Processed dataset and file path.

VISSSlib.distributions.addPerParticleVariables(level1dat_camAve, config)

Add per-particle variables to the dataset.

This function computes additional particle characteristics like complexity and normalized rime mass based on existing particle data.

Parameters

level1dat_camAvexarray.Dataset

Dataset containing particle data.

configdict

Configuration settings for computation.

Returns

xarray.Dataset

Dataset with additional particle variables.

VISSSlib.distributions.addVariables(calibDat, case, config, timeIndex, timeIndex1, sublevel, camera='leader')

Add additional variables to the calibrated dataset.

Computes derived quantities such as particle size distribution (PSD), moments, and quality flags based on the calibrated particle data.

Parameters

calibDatxarray.Dataset

Calibrated dataset with basic particle properties.

casestr

Case identifier for data context. For loopify decorator, this is a date string (YYYYMMDD) or date range (YYYYMMDD-YYYYMMDD) or comma-separated dates.

configdict

Configuration settings for computations.

timeIndexnumpy.ndarray

Time index for data aggregation.

timeIndex1numpy.ndarray

Detailed time index for computations.

sublevelstr

Processing sublevel. Must be one of [“match”, “track”, “detect”].

camerastr, optional

Camera identifier, defaults to “leader”. Used for match and track sublevels.

Returns

xarray.Dataset

Dataset with added variables.

VISSSlib.distributions.getPerTrackStatistics(level1dat, maxAngleDiff=20, extraVars=[])

Extract statistical information from particle tracks.

Computes track-level statistics by aggregating particle data along trajectory dimensions and calculating descriptive metrics.

Parameters

level1datxarray.Dataset

Input dataset containing particle tracking data.

maxAngleDifffloat, optional

Maximum angle difference for track continuation, defaults to 20.

extraVarslist, optional

Extra variables to include in computation, defaults to [].

Returns

tuple

Tuple containing track statistics dataset, 2D track data, time-resolved track data, individual particle data, and number of cuts.

VISSSlib.distributions.removeTrackEdges(level1dat_track2D, maxAngleDiff)

Remove track edges identified by large angular differences.

Identifies potentially erroneous track terminations by detecting large angular deviations and removes those sections from the data.

Parameters

level1dat_track2Dxarray.Dataset

Track dataset with particle trajectory data.

maxAngleDifffloat

Maximum allowable angular difference for track continuity.

Returns

tuple

Updated track dataset and number of removed edges.

VISSSlib.distributions.estimateObservationVolume(level1dat_time, config, DbinsPixel, timeIndex1)

Estimate observation volumes for different particle sizes.

Calculates the effective observation volume for each particle size bin based on camera configuration and particle characteristics.

Parameters

level1dat_timexarray.Dataset

Time-resolved dataset with camera rotation parameters.

configdict

Configuration settings for volume estimation.

DbinsPixelrange

Pixel bins for size distribution.

timeIndex1numpy.ndarray

Detailed time index for volume calculation.

Returns

list

List of estimated observation volumes for each bin.

VISSSlib.distributions.calibrateData(level2dat, level1dat_time, config, DbinsPixel, timeIndex1)

Apply calibration to level 2 data.

Transforms pixel-based measurements into calibrated physical units and adjusts for instrument-specific characteristics.

Parameters

level2datxarray.Dataset

Level 2 dataset with raw pixel measurements.

level1dat_timexarray.Dataset

Time-resolved dataset with camera parameters.

configdict

Configuration settings for calibration.

DbinsPixelrange

Pixel bins for size distribution.

timeIndex1numpy.ndarray

Detailed time index for calibration.

Returns

xarray.Dataset

Calibrated dataset with physical units.

VISSSlib.distributions.getDataQuality(case, config, timeIndex, timeIndex1, sublevel, camera=None)

Estimate comprehensive data quality for level 2 processing.

This function aggregates data quality metrics across both cameras for match and track processing levels.

Parameters

casestr

Case identifier for reference data. For loopify decorator, this is a date string (YYYYMMDD) or date range (YYYYMMDD-YYYYMMDD) or comma-separated dates.

configdict

Configuration settings for quality assessment.

timeIndexnumpy.ndarray

Time index for quality evaluation.

timeIndex1numpy.ndarray

Detailed time index for quality calculations.

sublevelstr

Processing sublevel. Must be one of [“match”, “track”, “detect”].

camerastr, optional

Camera identifier for quality metrics, defaults to None. For loopify_with_camera decorator, can be “all”, “leader”, or “follower”.

Returns

tuple

Quality metrics including recording failures, processing failures, blocked pixels ratios, blowing snow ratios, and observations ratio.

VISSSlib.distributions.createLeaderBox(width, height, delta=0, deltaExtra1=0, deltaExtra2=0)

Create a 3D box representing the leader camera observation volume.

Defines the physical volume observable by the leader camera based on image dimensions and boundary constraints.

Parameters

widthint

Image width.

heightint

Image height.

deltanumber, optional

Distance to all edges, defaults to 0.

deltaExtra1number, optional

Distance to left vertical edge, defaults to 0.

deltaExtra2number, optional

Distance to right vertical edge, defaults to 0.

Returns

trimesh.Trimesh

Trimesh object representing the leader observation volume.

VISSSlib.distributions.createFollowerBox(width, height, camera_phi, camera_theta, camera_Ofz, delta=0, deltaExtra1=0, deltaExtra2=0)

Create a 3D box representing the follower camera observation volume.

Defines the physical volume observable by the follower camera based on image dimensions, camera orientation, and boundary constraints.

Parameters

widthint

Image width.

heightint

Image height.

camera_phifloat

Roll of follower camera.

camera_thetafloat

Pitch of follower camera.

camera_Ofzfloat

Offset in z direction.

deltanumber, optional

Distance to all edges, defaults to 0.

deltaExtra1number, optional

Distance to left vertical edge, defaults to 0.

deltaExtra2number, optional

Distance to right vertical edge, defaults to 0.

Returns

trimesh.Trimesh

Trimesh object representing the follower observation volume.

VISSSlib.distributions.interpolateVolumes(Dfull, Dstep, volumes1)

Interpolate volumes considering the cubic dependency.

Applies cubic interpolation to volume data between specified intervals to obtain smooth volume estimates across all particle sizes.

Parameters

Dfullarray-like

Full range of particle sizes.

Dsteparray-like

Step sizes for interpolation.

volumes1array-like

Volume values at step sizes.

Returns

tuple

Interpolated size bins and volumes.

VISSSlib.files.findLastFile(config, prod, camera)

Find the last available file and related metadata for a given product and camera.

Parameters

configdict or str

Configuration settings or path to configuration file.

prodstr

Product name (e.g., ‘level1detect’).

camerastr

Camera identifier (e.g., ‘leader’ or ‘follower’).

Returns

tuple

Tuple containing: - foundLastFile: bool indicating if last file was found - lastCase: str, the last case found - lastFile: str, the path to the last file - lastFileTime: datetime obj, the timestamp of the last file

Notes

This function iterates through recent cases to find the most recent file of the requested product and camera combination.

class VISSSlib.files.FindFiles(case, camera, config, version='0.0')

Bases: object

Class to manage and locate files based on case and camera specifications.

This class handles finding and managing file paths for different processing levels of VISSS data, including level0, level1, level2, and level3 products. It provides methods to search, validate, and organize file locations based on timestamps and camera configurations.

Attributes

casestr

The case identifier (YYYYMMDD).

camerastr

The camera identifier (e.g., ‘leader’ or ‘follower’).

configdict

Configuration dictionary containing paths and settings.

versionstr

Version string for the VISSS processing.

yearstr

Year component of the case.

monthstr

Month component of the case.

daystr

Day component of the case.

hourstr

Hour component of the case (if applicable).

minutestr

Minute component of the case (if applicable).

datetimedatetime.datetime

Parsed datetime from the case identifier.

datetime64numpy.datetime64

Parsed datetime64 from the case identifier.

logpathstr

Path to log files for this camera.

outpathdict

Dictionary mapping processing level to output directory path.

fnamesPatterndict

Dictionary mapping processing level to filename pattern.

fnamesPatternExtdict

Dictionary mapping processing level to expanded filename pattern.

fnamesDailydict

Dictionary mapping processing level to daily filename pattern.

fnamesHourlydict

Dictionary mapping processing level to hourly filename pattern.

quicklookdict

Dictionary mapping processing level to quicklook image filename.

quicklookCurrentdict

Dictionary mapping processing level to current quicklook image filename.

quicklookPathdict

Dictionary mapping processing level to quicklook image path.

property yesterday

Get the previous day’s case identifier.

Returns

str

Date in YYYYMMDD format for the day before this case.

property yesterdayObject

Get the FindFiles object for the previous day.

Returns

FindFiles

FindFiles object for the previous day.

property tomorrow

Get the next day’s case identifier.

Returns

str

Date in YYYYMMDD format for the day after this case.

property tomorrowObject

Get the FindFiles object for the next day.

Returns

FindFiles

FindFiles object for the next day.

getEvents(skipExisting=True)

Retrieve and optionally create the event dataset for this case and camera.

Parameters

skipExistingbool, optional

Whether to skip processing if files already exist, defaults to True

Returns

tuple(str, xarray.Dataset)

Tuple of (event_filename, event_dataset) or (None, None) if not found

listFiles(level)

List all files of a specific processing level for this case and camera.

Parameters

levelstr

Processing level to list files for (e.g., ‘level1detect’).

Returns

list

List of full paths to files matching the pattern.

Raises

ValueError

If the level is not recognized.

listFilesExt(level, ignoreBrokenFiles=False)

List all files of a specific processing level for this case and camera, including broken and nodata files.

Parameters

levelstr

Processing level to list files for (e.g., ‘level1detect’).

ignoreBrokenFilesbool, optional

Ignore broken files in the listing (defaults to False).

Returns

list

List of full paths to files matching the pattern.

listBroken(level)

List all broken files of a specific processing level for this case and camera.

Parameters

levelstr

Processing level to list broken files for (e.g., ‘level1detect’).

Returns

list

List of full paths to broken files matching the pattern.

listNoData(level)

List all nodata files of a specific processing level for this case and camera.

Parameters

levelstr

Processing level to list nodata files for (e.g., ‘level1detect’).

Returns

list

List of full paths to nodata files matching the pattern.

listFilesWithNeighbors(level)

List all files of a specific processing level for this case and camera, including neighboring files.

Parameters

levelstr

Processing level to list files for (e.g., ‘level1detect’).

Returns

list

List of full paths to files (including neighbors) matching the pattern.

property isCompleteL0

Check if all level0 files for this case and camera are complete.

Returns

bool

True if all level0 files are complete, False otherwise.

property isCompleteL1detect

Check if all level1detect files for this case and camera are complete.

Returns

bool

True if all level1detect files are complete, False otherwise.

property isCompleteMetaFrames

Check if all metaFrames files for this case and camera are complete.

Returns

bool

True if all metaFrames files are complete, False otherwise.

property isCompleteL1match

Check if all level1match files for this case and camera are complete.

Returns

bool

True if all level1match files are complete, False otherwise.

property isCompleteL1track

Check if all level1track files for this case and camera are complete.

Returns

bool

True if all level1track files are complete, False otherwise.

isComplete(level, ignoreBrokenFiles=False, requireL0Files=False)

Check if all files of a specific processing level for this case and camera are complete.

Parameters

levelstr

Processing level to check (e.g., ‘level1detect’).

ignoreBrokenFilesbool, optional

Ignore broken files in the completeness check (defaults to False).

requireL0Filesbool, optional

Require the presence of level0 files (defaults to False).

Returns

bool

True if all files of the specified level are complete, False otherwise.

property nL0

Get the number of level0 files for this case and camera.

Returns

int

Number of level0 files.

property nMissingL0

Get the number of missing level0 files for this case and camera.

Returns

int

Number of missing level0 files.

property nMissingL1detect

Get the number of missing level1detect files for this case and camera.

Returns

int

Number of missing level1detect files.

property nMissingMetaFrames

Get the number of missing metaFrames files for this case and camera.

Returns

int

Number of missing metaFrames files.

property nMissingL1match

Get the number of missing level1match files for this case and camera.

Returns

int

Number of missing level1match files.

property nMissingL1track

Get the number of missing level1track files for this case and camera.

Returns

int

Number of missing level1track files.

nMissing(level, ignoreBrokenFiles=False, requireL0Files=False)

Get the number of missing files of a specific processing level for this case and camera.

Parameters

levelstr

Processing level to check (e.g., ‘level1detect’).

ignoreBrokenFilesbool, optional

Ignore broken files in the count (defaults to False).

requireL0Filesbool, optional

Require the presence of level0 files (defaults to False).

Returns

int

Number of missing files.

reportDuplicates(level)

Report duplicate files of a specific processing level for this case and camera.

Parameters

levelstr

Processing level to check (e.g., ‘level1detect’).

Returns

list

List of duplicate file paths.

class VISSSlib.files.Filenames(fname, config, version='0.0')

Bases: object

Class to manage file paths and related information for VISSS data processing.

This class provides methods to construct and manage file paths for different processing levels of VISSS data. It handles both level0 (raw video) and higher level files (processed data). The class supports operations such as finding neighboring files, determining if data is complete, and creating file paths for various processing stages.

Attributes

fnamedict

Dictionary mapping processing level to full file path.

configdict

Configuration dictionary containing paths and settings.

versionstr

Version string for the VISSS processing.

basenamestr

Basename of the file without extension.

dirnamestr

Directory name of the file.

casestr

Case identifier (YYYYMMDD).

yearstr

Year component of the case.

monthstr

Month component of the case.

daystr

Day component of the case.

timestampstr

Timestamp component of the case (HHMMSS).

datetimedatetime.datetime

Parsed datetime from the case identifier.

datetime64numpy.datetime64

Parsed datetime64 from the case identifier.

camerastr

Camera identifier (e.g., ‘leader_001’).

visssGenstr

VISSS generation identifier.

computerstr

Computer identifier.

basenameShortstr

Short basename for daily files.

outpathstr

Output path for the current date.

outpathDailystr

Daily output path for the current date.

logpathstr

Log path for this camera.

quicklookPathdict

Dictionary mapping processing level to quicklook image path.

property yesterday

Get the previous day’s case identifier.

Returns

str

Date in YYYYMMDD format for the day before this case.

property yesterdayObject

Get the FindFiles object for the previous day.

Returns

FindFiles

FindFiles object for the previous day.

property tomorrow

Get the next day’s case identifier.

Returns

str

Date in YYYYMMDD format for the day after this case.

property tomorrowObject

Get the FindFiles object for the next day.

Returns

FindFiles

FindFiles object for the next day.

filenamesOtherCamera(graceInterval=120, level='level1detect')

Find all relevant files of the other camera for a given level.

Parameters

graceIntervalint, optional

Grace interval in seconds to account for time offsets, defaults to 120.

levelstr, optional

Processing level to check, defaults to “level1detect”.

Returns

list

List of filenames from the other camera matching the criteria.

Notes

This method considers time windows and handles cases where files might be from adjacent days due to time synchronization issues.

property fnameTxtAllThreads

Find level 0 txt filenames of all threads.

Returns

dict

Dictionary mapping thread number to txt filename.

Notes

For single-threaded setups, this returns a dictionary with only thread 0. For multi-threaded setups, it returns file paths for all threads. Handles cases where files may have slightly different timestamps.

property fnameMovAllThreads

Find level 0 movie filenames of all threads.

Returns

dict

Dictionary mapping thread number to movie filename.

Notes

Similar to fnameTxtAllThreads but returns movie files instead of txt files. Uses the same logic for handling multiple threads and timestamp variations.

nextFile(level='level0', debug=False)

Find the next file in sequence for the specified processing level.

Parameters

levelstr, optional

Processing level, defaults to “level0”.

debugbool, optional

Enable debug output, defaults to False.

Returns

str or None

Path to the next file, or None if not found.

Notes

If the file does not exist yet, it searches across day boundaries for nearby files matching the processing level.

prevFile(level='level0', debug=False)

Find the previous file in sequence for the specified processing level.

Parameters

levelstr, optional

Processing level, defaults to “level0”.

debugbool, optional

Enable debug output, defaults to False.

Returns

str or None

Path to the previous file, or None if not found.

Notes

If the file does not exist yet, it searches across day boundaries for nearby files matching the processing level.

findNeighborFile(offset, level='level0', debug=False)

Find file at a specific offset from the current file.

Parameters

offsetint

Offset to search for (positive for future, negative for past).

levelstr, optional

Processing level, defaults to “level0”.

debugbool, optional

Enable debug output, defaults to False.

Returns

str or None

Path to the neighbor file, or None if not found.

Notes

Handles boundary conditions when searching across day boundaries.

nextFile2(level='level0', maxOffset=np.timedelta64(2, 'h'))

Alternative implementation to find next file using timestamp-based search.

Parameters

levelstr, optional

Processing level to search, defaults to “level0”.

maxOffsetnumpy.timedelta64, optional

Maximum time offset to search, defaults to 2 hours.

Returns

str or None

Path to the next file, or None if not found.

Notes

More robust than nextFile when the reference file does not exist yet. Uses timestamp comparisons to avoid issues with missing files.

prevFile2(level='level0', maxOffset=np.timedelta64(2, 'h'))

Alternative implementation to find previous file using timestamp-based search.

Parameters

levelstr, optional

Processing level to search, defaults to “level0”.

maxOffsetnumpy.timedelta64, optional

Maximum time offset to search, defaults to 2 hours.

Returns

str or None

Path to the previous file, or None if not found.

Notes

More robust than prevFile when the reference file does not exist yet. Uses timestamp comparisons to avoid issues with missing files.

getEvents(skipExisting=True)

Get (and create if necessary) event dataset for this case and camera.

Parameters

skipExistingbool, optional

Skip processing existing files (defaults to True).

Returns

tuple(str, xarray.Dataset)

Tuple containing event filename and event Dataset.

Notes

This method ensures event files are available for data validation and processing workflows.

class VISSSlib.files.FilenamesFromLevel(fname, config)

Bases: Filenames

Class to manage file paths for VISSS data processing starting from level 1 or level 2 files.

This class extends the Filenames class and is specifically designed to handle file paths when starting from processed level 1 or level 2 files. It extracts essential information from these files to reconstruct the full path structure including case, camera, and processing level details.

Attributes

fnamedict

Dictionary mapping processing level to full file path.

configdict

Configuration dictionary containing paths and settings.

versionstr

Version string for the VISSS processing.

basenamestr

Basename of the file without extension.

dirnamestr

Directory name of the file.

casestr

Case identifier (YYYYMMDD).

yearstr

Year component of the case.

monthstr

Month component of the case.

daystr

Day component of the case.

timestampstr

Timestamp component of the case (HHMMSS).

datetimedatetime.datetime

Parsed datetime from the case identifier.

datetime64numpy.datetime64

Parsed datetime64 from the case identifier.

camerastr

Camera identifier (e.g., ‘leader_001’).

visssGenstr

VISSS generation identifier.

computerstr

Computer identifier.

basenameShortstr

Short basename for daily files.

outpathstr

Output path for the current date.

outpathDailystr

Daily output path for the current date.

logpathstr

Log path for this camera.

quicklookPathdict

Dictionary mapping processing level to quicklook image path.

VISSSlib.fixes.fixMosaicTimeL1(dat1, config)

Attempt to fix drift of capture time with record_time.

This function attempts to correct timing drift between capture_time and record_time by estimating and interpolating drift patterns over time.

Parameters

dat1xarray.Dataset

Input dataset containing capture_time and record_time variables

configobject

Configuration object containing fps parameter for frame rate

Returns

xarray.Dataset

Dataset with corrected capture_time values

Notes

This is a poor attempt at fixing drift and is not used anymore. The function groups data into time chunks and estimates drift patterns to interpolate and correct the timing issues.

VISSSlib.fixes.captureIdOverflows(dat, config, storeOrig=True, idOffset=0, dim='pid')

Fix capture_id overflows for M1280 devices.

For M1280 devices, capture_id is a 16-bit integer that overflows every few minutes. This function detects and fixes overflow conditions by applying appropriate offsets.

Parameters

datxarray.Dataset

Input dataset containing capture_id and capture_time variables

configobject

Configuration object containing fps parameter for frame rate

storeOrigbool, optional

Whether to store original capture_id values, default is True

idOffsetint, optional

Constant offset to add to capture_id, default is 0

dimstr, optional

Dimension name for diff operations, default is “pid”

Returns

xarray.Dataset

Dataset with fixed capture_id values

Notes

This function handles the specific case where capture_id overflows due to being a 16-bit integer. It detects overflow points and applies corrections to maintain proper sequential numbering.

VISSSlib.fixes.revertIdOverflowFix(dat)

Revert capture_id overflow fix by restoring original values.

This function restores the original capture_id values by renaming the fixed and original variables back to their original names.

Parameters

datxarray.Dataset

Input dataset with fixed capture_id and capture_id_orig variables

Returns

xarray.Dataset

Dataset with original capture_id restored

Notes

This function is used to undo the effects of captureIdOverflows when needed for data recovery or analysis consistency.

VISSSlib.fixes.removeGhostFrames(metaDat, config, intOverflow=True, idOffset=0, fixIteration=3)

Remove ghost frames from MOSAiC follower data.

For MOSAiC follower devices, additional ghost frames are occasionally added to the dataset. These can be identified by their spacing being less than 1/fps apart. This function identifies and removes such frames.

Parameters

metaDatxarray.Dataset

Input dataset containing capture_time and capture_id variables

configobject

Configuration object containing fps parameter for frame rate

intOverflowbool, optional

Whether to handle integer overflows, default is True

idOffsetint, optional

Offset to add to capture_id, default is 0

fixIterationint, optional

Number of iterations to attempt ghost frame removal, default is 3

Returns

tuple

A tuple containing (fixed_dataset, dropped_frames, beyond_repair_flag) where: - fixed_dataset is the dataset with ghost frames removed - dropped_frames is the count of removed frames - beyond_repair_flag indicates if data is beyond repair

Notes

Ghost frames are typically identified by their spacing being significantly different from the expected 1/fps interval. The function performs multiple iterations to handle complex cases where ghost frames might be in data gaps.

VISSSlib.fixes.delayedClockReset(metaDat, config)

Check for and fix delayed clock reset issues.

This function detects delayed clock resets in the data and attempts to correct them by adjusting timestamps accordingly.

Parameters

metaDatxarray.Dataset

Input dataset containing capture_time and capture_id variables

configobject

Configuration object containing fps parameter for frame rate

Returns

xarray.Dataset

Dataset with corrected timestamps if reset was detected

Notes

Delayed clock resets are identified by large negative time differences (>10 seconds). The function handles both cases where integer overflows and timestamp issues coexist, and attempts to fix the timing problems by recalculating timestamps based on known good values.

VISSSlib.fixes.makeCaptureTimeEven(datF, config, dim='capture_time')

Make capture time even for M1280 follower devices.

For M1280 follower devices, significant drift can occur causing clocks to drift more than 1 frame apart within 10 minutes. This function creates a new time vector with even spacing based on a trusted capture_id.

Parameters

datFxarray.Dataset

Input dataset containing capture_time and capture_id variables

configobject

Configuration object containing fps parameter for frame rate

dimstr, optional

Dimension name for operations, default is “capture_time”

Returns

xarray.Dataset

Dataset with new evenly spaced capture_time_even variable

Notes

This function is specifically designed for capture_id offset estimation and creates a new time vector that maintains even spacing regardless of timing drift issues. It validates that the calculated slopes are within acceptable ranges.

VISSSlib.level3.retrieveM(y_obs, psd, air_temperature, Dmean, Dbound, frequency, config)

Perform optimal estimation retrieval for riming mass parameter.

This function uses the pyOptimalEstimation library to retrieve the riming mass parameter M from radar reflectivity observations and particle size distribution data.

Parameters

y_obsarray-like

Observed radar reflectivity values

psdarray-like

Particle size distribution data

air_temperaturefloat

Air temperature in Kelvin

Dmeanarray-like

Mean diameter values

Dboundarray-like

Diameter bin boundaries

frequencyfloat

Radar frequency in GHz

configobject

Configuration object containing retrieval settings

Returns

tuple

A tuple containing (M_oe, M_err, Ze_combinedRetrieval) where: - M_oe: Optimally estimated riming mass parameter - M_err: Error in the estimated riming mass - Ze_combinedRetrieval: Combined radar reflectivity retrieval

Notes

This function implements a Bayesian optimal estimation approach to retrieve the riming mass parameter from radar measurements and PSD data.

VISSSlib.level3.ssrga_parameter(M, elevation)

Calculate SSRGA parameters for given riming mass and elevation.

This function computes the SSRGA (Self-Similar Rayleigh-Gans Approximation) parameters for snowflake scattering based on riming mass and radar elevation.

Parameters

Mfloat or array-like

Riming mass parameter

elevationfloat

Radar elevation angle in degrees

Returns

tuple

A tuple containing (kappa, beta, gamma, zeta1, alpha_eff) parameters for SSRGA scattering calculations

Raises

ValueError

If elevation is not 90 or around 50 degrees

Notes

The SSRGA parameters are used in the scattering calculations for snowflakes with different riming characteristics at different radar elevations.

VISSSlib.level3.reflec_logM(X, ice, alt, temp, bins, dmean, dbound, shape, frequency, elevation)

Forward model for radar reflectivity based on riming mass.

This function calculates the radar reflectivity using a forward model that incorporates riming mass, particle size distribution, and atmospheric conditions.

Parameters

Xfloat

Logarithm of riming mass parameter

icearray-like

Ice particle size distribution

altfloat

Altitude in meters

tempfloat

Temperature in Kelvin

binsint

Number of size bins

dmeanarray-like

Mean diameter values

dboundarray-like

Diameter bin boundaries

shapestr

Snowflake habit shape

frequencyfloat

Radar frequency in GHz

elevationfloat

Radar elevation angle in degrees

Returns

float

Calculated radar reflectivity (Ze)

Notes

This function implements a forward model using pyPamtra to simulate radar reflectivity based on the physical properties of snowflakes with varying riming characteristics.

VISSSlib.level3.mass_size(M)

Interpolate mass-size relationship for riming parameter.

This function interpolates the mass-size relationship parameters (a and b) as a function of the riming mass parameter M using cubic spline interpolation.

Parameters

Mfloat or array-like

Riming mass parameter

Returns

tuple

A tuple containing (a_m, b_m) mass-size relationship parameters

Notes

This function implements a cubic spline interpolation based on lookup tables from Maherndl et al. (2023) for riming-dependent mass-size relationships.

VISSSlib.level3.dynamic_viscosity_air(temperature)

Calculate dynamic viscosity of dry air using Sutherland’s law.

This function computes the dynamic viscosity of dry air at a given temperature using the Sutherland’s law formula.

Parameters

temperaturefloat

Temperature in Kelvin

Returns

float

Dynamic viscosity of dry air in Pascal-seconds (Pa·s)

Notes

The calculation uses coefficients from F. M. White, Viscous Fluid Flow, 2nd ed., McGraw-Hill, (1991) and Kim et al., arXiv:physics/0410237v1.

VISSSlib.level3.dry_density_air(temperature, press)

Calculate dry air density using ideal gas law.

This function computes the density of dry air at given temperature and pressure using the ideal gas law.

Parameters

temperaturefloat

Temperature in Kelvin

pressfloat

Atmospheric pressure in Pascals

Returns

float

Dry air density in kg/m³

Notes

Uses the specific gas constant for dry air (R_s = 287.0500676 J/kg·K).

VISSSlib.level3.heymsfield10_particles_M(Dmax, M, temperature, press, shape)

Calculate fall velocity for snowflakes using Heymsfield et al. (2010) parameterization.

This function computes the fall velocity of snowflakes using the Heymsfield et al. (2010) parameterization that depends on riming mass, particle size, and atmospheric conditions.

Parameters

Dmaxfloat or array-like

Maximum particle diameter

Mfloat or array-like

Riming mass parameter

temperaturefloat

Temperature in Kelvin

pressfloat

Atmospheric pressure in Pascals

shapestr

Snowflake habit shape

Returns

float or array-like

Fall velocity in m/s

Notes

This implementation follows the Heymsfield et al. (2010) approach for calculating fall velocities of rimed snowflakes based on their mass and size characteristics.

VISSSlib.level3.retrieveCombinedRiming(case, config, skipExisting=True, writeNc=True, doQuicklook=True)

Apply combined riming retrieval to VISSS data.

This function performs a combined riming retrieval following the methodology described in Maherndl et al. (2023) to determine riming mass and related microphysical parameters from radar and meteorological data.

Parameters

casestr

Case identifier for data processing

configobject or str

Configuration object or path to configuration file

skipExistingbool, optional

Skip processing if output file exists, default is True

writeNcbool, optional

Write NetCDF output file, default is True

doQuicklookbool, optional

Generate quicklook plots, default is True

Returns

tuple

A tuple containing (lv3Dat, lv3File) where: - lv3Dat: Processed level 3 dataset - lv3File: Output file path

Notes

This function implements the combined riming retrieval algorithm that: 1. Retrieves riming mass parameter using optimal estimation 2. Derives microphysical parameters from the retrieved mass 3. Calculates ice water content and snowfall rates 4. Generates NetCDF output and quicklook plots

References

Maherndl, N., M. Maahn, F. Tridon, J. Leinonen, D. Ori, and S. Kneifel, 2023: A riming-dependent parameterization of scattering by snowflakes using the self-similar rayleigh–gans approximation. Q. J. R. Meteorolog. Soc., 149, 3562–3581, doi:10.1002/qj.4573.

VISSSlib.matching.rotate_L2F(L_x, L_y, L_z, phi, theta, psi)

Rotate from leader to follower coordinate system.

Parameters

L_xfloat

Leader x coordinate (in common xyz)

L_yfloat

Leader y coordinate (in common xyz)

L_zfloat

Leader z coordinate (in common xyz)

phifloat

Follower roll in degrees

thetafloat

Follower pitch in degrees

psifloat

Follower yaw in degrees

Returns

tuple

Follower x, y, z coordinates

VISSSlib.matching.shiftRotate_L2F(L_x, L_y, L_z, phi, theta, psi, Olx, Ofy, Ofz)

Shift and rotate from leader to follower coordinate system.

Parameters

L_xfloat

Leader x coordinate (in common xyz)

L_yfloat

Leader y coordinate (in common xyz)

L_zfloat

Leader z coordinate (in common xyz)

phifloat

Follower roll in degrees

thetafloat

Follower pitch in degrees

psifloat

Follower yaw in degrees

Olxfloat

Leader shift in x direction

Ofyfloat

Follower shift in y direction

Ofzfloat

Follower shift in z direction

Returns

tuple

Follower x, y, z coordinates

VISSSlib.matching.rotate_F2L(F_xp, F_yp, F_zp, phi, theta, psi)

Rotate from follower to leader coordinate system.

Parameters

F_xpfloat

Follower x coordinate (in common xyz)

F_ypfloat

Follower y coordinate (in common xyz)

F_zpfloat

Follower z coordinate (in common xyz)

phifloat

Follower roll in degrees

thetafloat

Follower pitch in degrees

psifloat

Follower yaw in degrees

Returns

tuple

Leader x, y, z coordinates

VISSSlib.matching.shiftRotate_F2L(F_x, F_y, F_z, phi, theta, psi, Olx, Ofy, Ofz)

Shift and rotate from follower to leader coordinate system.

Parameters

F_xfloat

Follower x coordinate (in common xyz)

F_yfloat

Follower y coordinate (in common xyz)

F_zfloat

Follower z coordinate (in common xyz)

phifloat

Follower roll in degrees

thetafloat

Follower pitch in degrees

psifloat

Follower yaw in degrees

Olxfloat

Leader shift in x direction

Ofyfloat

Follower shift in y direction

Ofzfloat

Follower shift in z direction

Returns

tuple

Leader x, y, z coordinates

VISSSlib.matching.calc_L_z(L_x, F_yp, F_zp, phi, theta, psi)

Estimate z coordinate for the leader based on combined leader and follower measurements.

Parameters

L_xfloat

x measurement of leader

F_ypfloat

y measurement of follower without shift

F_zpfloat

z measurement of follower without shift

phifloat

Follower roll in degrees

thetafloat

Follower pitch in degrees

psifloat

Follower yaw in degrees

Returns

float

z coordinate as seen by leader ignoring offsets

VISSSlib.matching.calc_L_z_withOffsets(L_x, F_y, F_z, camera_phi=0, camera_theta=0, camera_psi=0, camera_Ofy=0, camera_Ofz=0, camera_Olx=0)

Estimate z coordinate for the leader based on combined leader and follower measurements.

Parameters

L_xfloat

x measurement of leader

F_yfloat

y measurement of follower

F_zfloat

z measurement of follower

camera_phifloat, optional

Follower roll in degrees (default is 0)

camera_thetafloat, optional

Follower pitch in degrees (default is 0)

camera_psifloat, optional

Follower yaw in degrees (default is 0)

camera_Ofyfloat, optional

Follower shift in y direction (default is 0)

camera_Ofzfloat, optional

Follower shift in z direction (default is 0)

camera_Olxfloat, optional

Leader shift in x direction (default is 0)

Returns

float

z coordinate as seen by leader

VISSSlib.matching.forward(x, L_x=None, F_y=None, F_z=None)

Forward model for pyOptimalEstimation.

Parameters

xpandas Series

State vector “phi”, “theta”, “psi”, “Ofy”, “Ofz”, “Olx”

L_xarray, optional

x coordinate as seen by the leader (default is None)

F_yarray, optional

y coordinate as seen by the follower (default is None)

F_zarray, optional

z coordinate as seen by the follower (default is None)

Returns

pandas Series

z coordinate as seen by leader

VISSSlib.matching.retrieveRotation(dat3, x_ap, x_cov_diag, y_cov_diag, config, verbose=False, maxIter=30)

Apply Optimal Estimation to retrieve rotation of cameras.

Parameters

dat3xarray Dataset

Input data containing particle information

x_apdict

A priori values for state variables

x_cov_diagarray-like

Diagonal elements of a priori covariance matrix

y_cov_diagarray-like

Diagonal elements of observation covariance matrix

configobject

Configuration object with camera settings

verbosebool, optional

Whether to print verbose output (default is False)

maxIterint, optional

Maximum number of iterations (default is 30)

Returns

tuple

(x_op, x_op_err, dgf_x) - Optimal state vector, errors, and goodness of fit

VISSSlib.matching.probability(x, mu, sigma, delta)

Calculate probability using normal distribution.

Parameters

xarray-like

Values to calculate probability for

mufloat

Mean of the distribution

sigmafloat

Standard deviation of the distribution

deltafloat

Width of integration interval

Returns

array-like

Probability values

VISSSlib.matching.step(x, mu, sigma)

Step function for probability calculation.

Parameters

xarray-like

Values to calculate probability for

mufloat

Mean value for comparison

sigmafloat

Threshold for comparison

Returns

array-like

Binary values (0 or 1) based on comparison

VISSSlib.matching.removeDoubleCounts(mPart, mProp, doubleCounts)

Remove duplicate particle matches.

Parameters

mPartarray-like

Particle match indices

mProparray-like

Match probabilities

doubleCountsarray-like

Indices of particles that appear multiple times

Returns

tuple

Updated mPart and mProp arrays with duplicates removed

VISSSlib.matching.doMatch(leader1D, follower1D, sigmaIn, mu, delta, config, rotate, ptpTime, minProp=1e-10, minNumber4Stats=10, maxMatches=100, indexOffset=0, testing=False)

Match particles between leader and follower cameras.

Parameters

leader1Dxarray Dataset

Leader camera particle data

follower1Dxarray Dataset

Follower camera particle data

sigmaIndict or str

Sigma values for matching criteria or ‘default’

mudict

Mean values for matching criteria

deltadict

Delta values for matching criteria

configobject

Configuration object with camera settings

rotatedict

Rotation parameters

ptpTimebool

Whether to use PTP time synchronization

minPropfloat, optional

Minimum required probability (default is 1e-10)

minNumber4Statsint, optional

Minimum number of samples for statistics (default is 10)

maxMatchesint, optional

Maximum number of matches to consider (default is 100)

indexOffsetint, optional

Offset for pairing indices (default is 0)

testingbool, optional

Whether to generate test plots (default is False)

Returns

tuple

(matchedDat, disputedPairs, new_sigma, new_mu) - Matched data, disputed pairs, updated sigma and mu values

VISSSlib.matching.get3DPosition(leaderDat, followerDat, config)

Get 3D positions from leader and follower data.

Parameters

leaderDatxarray Dataset

Leader camera particle data

followerDatxarray Dataset

Follower camera particle data

configobject

Configuration object with camera settings

Returns

tuple

(L_x, L_z, F_y, F_z) - Position coordinates

VISSSlib.matching.get3DCentroid(leaderDat, followerDat, config)

Get 3D centroids from leader and follower data.

Parameters

leaderDatxarray Dataset

Leader camera particle data

followerDatxarray Dataset

Follower camera particle data

configobject

Configuration object with camera settings

Returns

tuple

(L_x, L_z, F_y, F_z) - Centroid coordinates

VISSSlib.matching.addPosition(matchedDat, rotate, rotate_err, config)

Add position variable to match dataset based on retrieved rotation parameters.

Parameters

matchedDatxarray Dataset

Matched particle data

rotatedict

Rotation parameters

rotate_errdict

Rotation parameter errors

configobject

Configuration object with camera settings

Returns

xarray Dataset

Updated dataset with position information added

VISSSlib.matching.doMatchSlicer(leader1D, follower1D, sigma, mu, delta, config, rotate, ptpTime, minProp=1e-10, maxMatches=100, minNumber4Stats=10, chunckSize=700, testing=False)

Do matching with slicing to handle memory constraints.

Parameters

leader1Dxarray Dataset

Leader camera particle data

follower1Dxarray Dataset

Follower camera particle data

sigmadict

Sigma values for matching criteria

mudict

Mean values for matching criteria

deltadict

Delta values for matching criteria

configobject

Configuration object with camera settings

rotatedict

Rotation parameters

ptpTimebool

Whether to use PTP time synchronization

minPropfloat, optional

Minimum required probability (default is 1e-10)

maxMatchesint, optional

Maximum number of matches to consider (default is 100)

minNumber4Statsint, optional

Minimum number of samples for statistics (default is 10)

chunckSizeint, optional

Size of data chunks (default is 700)

testingbool, optional

Whether to generate test plots (default is False)

Returns

tuple

(matchedDat, disputedPairs, new_sigma, new_mu) - Matched data, disputed pairs, updated sigma and mu values

VISSSlib.matching.matchParticles(fnameLv1Detect, config, y_cov_diag=2.7224999999999997, version='0.0', chunckSize=1000, rotate='config', rotate_err='config', maxDiffMs='config', rotationOnly=False, nPoints=500, sigma='default', nSamples4rot=300, minSamples4rot=100, testing=False, minDMax4rot=0, singleParticleFramesOnly=False, doRot=False, writeNc=True, offsetsOnly=False, subset=None, maxIter=30, skipExisting=True)

Match particles between leader and follower cameras.

Parameters

fnameLv1Detectstr

Path to level1 detect file

configobject

Configuration object with camera settings

y_cov_diagfloat, optional

Observation covariance diagonal (default is 1.65**2)

versionstr, optional

Version string (default is __version__)

chunckSizeint, optional

Size of data chunks (default is 1000)

rotatedict or str, optional

Initial rotation parameters or “config” (default is “config”)

rotate_errdict or str, optional

Initial rotation parameter errors or “config” (default is “config”)

maxDiffMsfloat or str, optional

Maximum time difference in milliseconds or “config” (default is “config”)

rotationOnlybool, optional

Whether to only estimate rotation (default is False)

nPointsint, optional

Number of points for estimation (default is 500)

sigmadict or str, optional

Sigma values for matching criteria or “default” (default is “default”)

nSamples4rotint, optional

Number of samples for rotation estimation (default is 300)

minSamples4rotint, optional

Minimum samples for rotation estimation (default is 100)

testingbool, optional

Whether to generate test plots (default is False)

minDMax4rotfloat, optional

Minimum DMax threshold for rotation estimation (default is 0)

singleParticleFramesOnlybool, optional

Whether to use only single particle frames (default is False)

doRotbool, optional

Whether to perform rotation estimation (default is False)

writeNcbool, optional

Whether to write NetCDF files (default is True)

offsetsOnlybool, optional

Whether to only estimate offsets (default is False)

subsettuple, optional

Subset of particles to process (default is None)

maxIterint, optional

Maximum iterations for optimization (default is 30)

skipExistingbool, optional

Whether to skip existing files (default is True)

Returns

tuple

(fname1Match, matchedDat, rotate_final, rotate_err_final, nLeader, nFollower, nMatched, errors) - Results of matching

VISSSlib.matching.createMetaRotation(case, config, skipExisting=True, version='0.0', y_cov_diag=2.7224999999999997, chunckSize=1000, rotate='config', rotate_err='config', maxDiffMs='config', nPoints=500, sigma='default', minDMax4rot=10, nSamples4rot=300, minSamples4rot=50, testing=False, completeDaysOnly=True, writeNc=True, stopOnFailure=False, maxAgeDaysPrevFile=1, doPlots=True)

Create meta rotation data for camera alignment.

Parameters

casestr

Case identifier

configobject

Configuration object with camera settings

skipExistingbool, optional

Whether to skip existing files (default is True)

versionstr, optional

Version string (default is __version__)

y_cov_diagfloat, optional

Observation covariance diagonal (default is 1.65**2)

chunckSizeint, optional

Size of data chunks (default is 1000)

rotatedict or str, optional

Initial rotation parameters or “config” (default is “config”)

rotate_errdict or str, optional

Initial rotation parameter errors or “config” (default is “config”)

maxDiffMsfloat or str, optional

Maximum time difference in milliseconds or “config” (default is “config”)

nPointsint, optional

Number of points for estimation (default is 500)

sigmadict or str, optional

Sigma values for matching criteria or “default” (default is “default”)

minDMax4rotfloat, optional

Minimum DMax threshold for rotation estimation (default is 10)

nSamples4rotint, optional

Number of samples for rotation estimation (default is 300)

minSamples4rotint, optional

Minimum samples for rotation estimation (default is 50)

testingbool, optional

Whether to generate test plots (default is False)

completeDaysOnlybool, optional

Whether to process only complete days (default is True)

writeNcbool, optional

Whether to write NetCDF files (default is True)

stopOnFailurebool, optional

Whether to stop on failure (default is False)

maxAgeDaysPrevFileint, optional

Maximum age of previous file in days (default is 1)

Returns

tuple

(metaRotation, fnameMetaRotation) - Meta rotation data and file name

VISSSlib.matching.manualRotationEstimate(cases, settings, nPoints=1000, iterations=4, minSamples4rot=90)

Estimate camera rotation parameters through iterative particle matching.

This function processes multiple cases to estimate optimal camera rotation parameters by iteratively refining the rotation estimate using particle matching. Each case undergoes up to 4 iterations of matching with progressively refined parameters.

Parameters

caseslist of str

List of case identifiers to process in format [“YYYYMMDD-HHMMSS”]

settingsdict or str

Configuration settings or path to settings file

nPointsint, optional

Number of points to use in matching (default=1000)

iterationsint, optional

Number of iterations (default=4)

Returns

str

yaml Dictionary with case names as keys and rotation parameters as values Format: {case: {“transformation”: dict, “transformation_err”: dict}}

Notes

The function performs these steps for each case: 1. Load level1 detection data 2. Calculate minimum particle size (minSize) for filtering 3. Run up to 4 iterations of particle matching:

• 1st iteration: Full parameter set with strict filters

• Subsequent iterations: Relaxed parameters using previous rotation estimate

4. Validate results at each iteration

5. Store final rotation parameters if all validations pass

VISSSlib.metadata.getMetaData(fnames, camera, config, stopAfter=-1, testMovieFile=False, includeHeader=False, idOffset=0, fixIteration=3)

Get metadata from video files for a given camera.

This function processes metadata files from video recordings and extracts timing and capture information for each frame. It handles various issues such as timestamp jumps, ghost frames, and capture_id overflows.

Parameters

fnamesstr or list

Filename or list of filenames for metadata files

camerastr

Camera identifier

configdict

Configuration dictionary containing processing parameters

stopAfterint, optional

Stop processing after specified number of frames, default is -1 (all)

testMovieFilebool, optional

Whether to test movie files for integrity, default is False

includeHeaderbool, optional

Whether to include header information, default is False

idOffsetint, optional

Offset for capture IDs, default is 0

fixIterationint, optional

Iteration count for fixing issues, default is 3

Returns

tuple

Tuple containing (metaDat, droppedFrames, beyondRepair) where: - metaDat: xarray Dataset with metadata or None if no data - droppedFrames: number of frames dropped due to issues - beyondRepair: boolean indicating if data is beyond repair

VISSSlib.metadata.readHeaderData(fname, returnLasttime=False)

Read header information from metadata file.

This function parses the header section of a metadata file to extract configuration and timing information.

Parameters

fnamestr

Path to the metadata file

returnLasttimebool, optional

Whether to return last timestamp information, default is False

Returns

tuple

Tuple containing header information: - record_starttime: datetime of record start - asciiVersion: ASCII file version - gitTag: Git tag - gitBranch: Git branch - capture_starttime: datetime of capture start - capture_firsttime: datetime of first capture - capture_lasttime: datetime of last capture (if returnLasttime=True) - last_id: last capture ID (if returnLasttime=True) - serialnumber: camera serial number - configuration: camera configuration - hostname: host machine name - cameraTemperature: camera temperature - transferQueueCurrentBlockCount: current block count in transfer queue - transferMaxBlockSize: maximum block size in transfer queue - ptpStatus: PTP status

VISSSlib.metadata.createMetaFrames(case, camera, config, skipExisting=True, writeNc=True, doPlot=True)

Create metadata frames for a given case and camera.

This function processes all metadata files for a given case and camera to generate consolidated metadata frames for further processing.

Parameters

casestr

Case identifier

camerastr

Camera identifier

configdict or str

Configuration dictionary or path to configuration file

skipExistingbool, optional

Whether to skip existing files, default is True

doPlotbool, optional

Do plot, default is True

Returns

xarray.Dataset or None

Dataset with metadata or None if no data

VISSSlib.metadata.getEvents(fnames0, config, fname0status=None)

Get event metadata for a given set of files.

This function processes metadata files to extract event information including file start times, capture times, and camera status.

Parameters

fnames0list

List of metadata file names

configdict

Configuration dictionary

fname0statusstr, optional

Status file name, default is None

Returns

xarray.Dataset

Dataset containing event metadata

VISSSlib.metadata.createEvent(case, camera, config, skipExisting=True, quiet=False, writeNc=True, version='0.0')

Create event file for a given case and camera.

This function generates event metadata files that summarize the status and timing information for all video files in a given case.

Parameters

casestr

Case identifier

camerastr

Camera identifier

configdict or str

Configuration dictionary or path to configuration file

skipExistingbool, optional

Whether to skip existing files, default is True

quietbool, optional

Whether to suppress logging output, default is False

versionstr, optional

Version string, default is __version__

Returns

xarray.Dataset or None

Dataset with event metadata or None if skipped

class VISSSlib.products.DataProduct(level, case, settings, fileQueue, camera, relatives=None, addRelatives=True, childrensRelatives={})

Bases: object

addRelatives()

Add relative products to the current product.

This method recursively adds parent products based on the parent names defined for the current level.

generateAllCommands(skipExisting=True, withParents=True)

Generate all commands for processing this product and its dependencies.

Parameters

skipExistingbool, default True

Whether to skip existing files

withParentsbool, default True

Whether to include parent commands

Returns

list

List of commands to execute

generateCommands(skipExisting=True, nCPU=1, bin=None)

Generate commands for processing this product.

Parameters

skipExistingbool, default True

Whether to skip existing files

nCPUint, default 1

Number of CPU cores to use

binstr, optional

Python binary path

Returns

list

List of commands to execute

Raises

ValueError

If the level is not recognized

submitCommands(skipExisting=True, checkForDuplicates=False, withParents=True, runWorkers=False)

Submit commands to the task queue.

Parameters

skipExistingbool, default True

Whether to skip existing files

checkForDuplicatesbool, default False

Whether to check for duplicate commands

withParentsbool, default True

Whether to include parent commands

runWorkersbool, default False

Whether to run workers immediately

runWorkers(nJobs=2, waitTime=1)

Run worker processes.

Parameters

nJobsint, default os.cpu_count()

Number of jobs to run

deleteQueue()

Delete all tasks from the queue.

property isComplete

Check if all required files for this level exist.

Returns

bool

True if all files are complete, False otherwise

property fileCreation

Get the creation time of this product.

Returns

float

Modification time of the newest file

property parentsComplete

Check if product’s parent are complete.

Returns

bool

True if all parents are complete, False otherwise

report(withParents=True)

Print a report about this product’s status.

Parameters

withParentsbool, default True

Whether to include parent reports

property dataAvailable

Check if data is available for this product.

Returns

bool

True if data is available, False otherwise

property allComplete

Check if this product and all its dependencies are complete.

Returns

bool

True if all is complete, False otherwise

property nFiles

Get the number of files for this product.

Returns

int

Number of files

listFilesExt()

List all files for this product.

Returns

list

List of file paths

listFiles()

List files for this product.

Returns

list

List of file paths

listBroken()

List broken files for this product.

Returns

list

List of broken file paths

listNoData()

List files with no data for this product.

Returns

list

List of no-data file paths

cleanUpBroken(withParents=False, withNoData=False)

Clean up broken files.

Parameters

withParentsbool, default False

Whether to clean up parents too

withNoDatabool, default False

Whether to clean up no-data files too

cleanUpDuplicates(withParents=False)

Clean up duplicate files.

Parameters

withParentsbool, default False

Whether to clean up parents too

class VISSSlib.products.allDone(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.level2track(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.level2match(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.level2detect(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.level1track(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.level1match(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.metaRotation(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.level1detect(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.metaFrames(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.metaEvents(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.level0(case, settings, fileQueue, camera='leader')

Bases: DataProduct

class VISSSlib.products.DataProductRange(level, nDays, settings, fileQueue, camera='leader', addRelatives=True)

Bases: object

generateAllCommands(skipExisting=True, withParents=True)

Generate all commands for the range of products.

Parameters

skipExistingbool, default True

Whether to skip existing files

withParentsbool, default True

Whether to include parent commands

Returns

list

List of commands to execute

submitCommands(skipExisting=True, checkForDuplicates=False, withParents=True, runWorkers=False)

Submit commands for the range of products.

Parameters

skipExistingbool, default True

Whether to skip existing files

checkForDuplicatesbool, default False

Whether to check for duplicate commands

withParentsbool, default True

Whether to include processing of product’s parents

runWorkersbool, default False

Whether to run workers immediately

property isComplete

Check if all products in the range are complete.

Returns

bool

True if all products are complete, False otherwise

runWorkers(nJobs=2, waitTime=1)

Run worker processes for the range.

Parameters

nJobsint, default os.cpu_count()

Number of jobs to run

deleteQueue()

Delete all tasks from the queue.

cleanUpBroken(withParents=False, withNoData=False)

Clean up broken files for the range.

Parameters

withParentsbool, default False

Whether to clean up parents too

withNoDatabool, default False

Whether to clean up no-data files too

cleanUpDuplicates(withParents=False)

Clean up duplicate files for the range.

Parameters

withParentsbool, default False

Whether to clean up parents too

VISSSlib.products.submitAll(case, settings, fileQueue, doMetaRot=True, submitJobs=True, skipExisting=True, checkForDuplicates=True, runWorkers=False, cleanUpBroken=False, cleanUpDuplicates=False)

Submit all processing jobs of for a given range of days. All processing levels are considered if corresponding input files are available

Parameters

casestr

Case or case range identifier for the data to process

settingsstr

Path to settings file

fileQueuestr

File queue for task management

doMetaRotbool, default True

Whether to perform meta rotation

submitJobsbool, default True

Whether to submit jobs to the queue

skipExistingbool, default True

Whether to skip existing files

checkForDuplicatesbool, default True

Whether to check for duplicate commands

runWorkersbool, default False

Whether to run workers immediately

cleanUpBrokenbool, default False

Whether to clean up broken files

cleanUpDuplicatesbool, default False

Whether to clean up duplicate files

Returns

object

DataProductRange object

VISSSlib.products.processAll(case, config, ignoreErrors=False, nJobs=<built-in function cpu_count>, fileQueue=None, skipExisting=True)

Process VISSS data for a specific case across all processing levels.

This function orchestrates the complete processing pipeline for a given case, handling both leader and follower cameras where applicable. It processes through various levels of data products including metadata creation, particle detection, matching, tracking, and level 2/3 retrievals.

Parameters

casestr

Case or case range identifier for the data to process

configstr or object

Configuration settings for processing. Can be a path to a settings file or a configuration object

ignoreErrorsbool, default False

If True, continue processing even if errors occur in individual steps

nJobsint or callable, default os.cpu_count

Number of parallel jobs to run. If callable, it will be called to get the number of jobs. This parameter is passed to the workers function.

fileQueuestr, optional

File queue for task management. If None, a temporary queue will be created.

skipExistingbool, default True

Whether to skip existing files during processing

Notes

The actual processing flow is:

1. Meta Events creation

2. Level 1 detection

3. Meta Rotation (if enabled)

4. Level 1 matching

5. Level 1 tracking

6. Level 2 matching

7. Level 2 tracking

8. Level 2 detection (if enabled)

9. Level 3 combined riming retrieval (if enabled)

10. All Done marker

For each processing level, both leader and follower cameras are processed where applicable. The function also handles error checking to ensure successful completion of each stage.

Note that this is a rather unefficient way of processing the data. It is recommended to use submitAll and run the workers separately

VISSSlib.products.processRealtime(case, settings, skipExisting=True)

Process VISSS data products that do not require significant computing resources for a specific case or case range. Calls * metadata.createEvent * quicklooks.level0Quicklook * metadata.createMetaFrames * tools.reportLastFiles

Parameters

casestr

Case identifier for the data to process

settingsstr

Path to settings file

skipExistingbool, default True

Whether to skip existing files during processing

Notes

The processing sequence includes: 1. Creating metadata events 2. Generating level 0 quicklooks 3. Creating metadata frames 4. Reporting last processed files

This function is designed for real-time processing scenarios where immediate results are needed.

VISSSlib.quicklooks.loop(level, nDays, settings, version='0.0', skipExisting=True, endYesterday=False)

Generate quicklooks for a specified level over a range of days.

Parameters

levelstr

The data level for which to generate quicklooks. Options include: ‘level0’, ‘level1detect’, ‘metaFrames’, ‘level2detect’, ‘level1match’, ‘level2match’, ‘level2track’, ‘level3combinedRiming’

nDaysint

Number of days to process, starting from today

settingsstr

Path to the configuration settings file

versionstr, optional

Version identifier for the processing, by default __version__

skipExistingbool, optional

Whether to skip generation if output already exists, by default True

endYesterdaybool, optional

Whether to end the case range at yesterday. Default is True. This parameter is passed to getCaseRange.

Returns

None

Function processes files but doesn’t return anything directly

Raises

ValueError

If the specified level is not recognized

VISSSlib.quicklooks.createLevel1detectQuicklookHourly(case, camera, config, hours=range(1, 24), version='0.0', container_width=200, container_height_max=600, nTiles=120, nRows=4, extra=1, readParticlesFromFiles=True, skipExisting=True, ffOut='default', timeStep='fixed', minBlur='config', minSize=8, omitLabel4small='config', timedelta=np.timedelta64(1, 'h'))

Create hourly version of particle quicklook.

Parameters

casestr

Date string in format YYYYMMDD

camerastr

Camera identifier

configdict

Configuration dictionary

hoursrange, optional

Hours to process, by default range(1, 24)

versionstr, optional

Version identifier, by default __version__

container_widthint, optional

Width of each tile container, by default 200

container_height_maxint, optional

Maximum height of each tile container, by default 600

nTilesint, optional

Number of tiles per row, by default 120

nRowsint, optional

Number of rows in output, by default 4

extraint, optional

Extra spacing between tiles, by default 1

readParticlesFromFilesbool, optional

Whether to read particles from files, by default True

skipExistingbool, optional

Whether to skip if output exists, by default True

ffOutstr, optional

Output file path, by default “default”

timeStepstr, optional

Time step method (‘fixed’ or ‘variable’), by default “fixed”

minBlurfloat or str, optional

Minimum blur threshold, by default “config”

minSizeint or str, optional

Minimum particle size, by default 8

omitLabel4smallbool or str, optional

Whether to omit labels for small particles, by default “config”

timedeltanumpy.timedelta64, optional

Time window for data selection, by default np.timedelta64(1, “h”)

Returns

None

Function processes files but doesn’t return anything directly

VISSSlib.quicklooks.createLevel1detectQuicklook(timestamp, camera, config, version='0.0', container_width=200, container_height_max=300, nTiles=60, nRows=4, extra=1, readParticlesFromFiles=True, skipExisting=True, ffOut='default', timeStep='variable', minBlur='config', minSize='config', omitLabel4small='config', timedelta=np.timedelta64(1, 'D'), returnFig=True)

Create level1detect quicklook for a given timestamp.

Parameters

timestampstr

Timestamp string in format YYYYMMDDHH or YYYYMMDD

camerastr

Camera identifier

configdict

Configuration dictionary

versionstr, optional

Version identifier, by default __version__

container_widthint, optional

Width of each tile container, by default 200

container_height_maxint, optional

Maximum height of each tile container, by default 300

nTilesint, optional

Number of tiles per row, by default 60

nRowsint, optional

Number of rows in output, by default 4

extraint, optional

Extra spacing between tiles, by default 1

readParticlesFromFilesbool, optional

Whether to read particles from files, by default True

skipExistingbool, optional

Whether to skip if output exists, by default True

ffOutstr, optional

Output file path, by default “default”

timeStepstr, optional

Time step method (‘fixed’ or ‘variable’), by default “variable”

minBlurfloat or str, optional

Minimum blur threshold, by default “config”

minSizeint or str, optional

Minimum particle size, by default “config”

omitLabel4smallbool or str, optional

Whether to omit labels for small particles, by default “config”

timedeltanumpy.timedelta64, optional

Time window for data selection, by default np.timedelta64(1, “D”)

returnFigbool, optional

Whether to return the figure, by default True

Returns

tuple

(output_file_path, figure) if returnFig=True, otherwise just output_file_path

class VISSSlib.quicklooks.Packer_patched(images)

Bases: Packer

Patched image_packer routine that works without files.

This class patches the image_packer library to work with in-memory images rather than requiring temporary files.

Attributes

_uid_to_filepathdict

Mapping of unique identifiers to image objects

_pieceslist

List of image pieces to pack

_has_alphabool

Flag indicating whether images have alpha channels

pack(container_width, options=None, container_height_max=100)

Packs multiple images of different sizes or formats into one image.

Parameters

container_widthint

Width of the container image

optionsdict, optional

Packing options, by default None

container_height_maxint, optional

Maximum height of the container, by default 100

Returns

PIL.Image.Image

Packed image

VISSSlib.quicklooks.level0Quicklook(case, camera, config, version='0.0', skipExisting=True)

Create level0 quicklook for a given case and camera.

Parameters

casestr

Date string in format YYYYMMDD

camerastr

Camera identifier

configdict

Configuration dictionary

versionstr, optional

Version identifier, by default __version__

skipExistingbool, optional

Whether to skip if output exists, by default True

Returns

tuple

(output_file_path, None) if successful, (None, None) otherwise

VISSSlib.quicklooks.metaFramesQuicklook(case, camera, config, version='0.0', skipExisting=True, plotCompleteOnly=True)

Create metaFrames quicklook for a given case and camera.

Parameters

casestr

Date string in format YYYYMMDD

camerastr

Camera identifier

configdict

Configuration dictionary

versionstr, optional

Version identifier, by default __version__

skipExistingbool, optional

Whether to skip if output exists, by default True

plotCompleteOnlybool, optional

Whether to only plot if data is complete, by default True

Returns

tuple

(output_file_path, matplotlib.figure.Figure) if successful, (None, None) otherwise

VISSSlib.quicklooks.createLevel1matchQuicklook(case, config, skipExisting=True, version='0.0', plotCompleteOnly=True, returnFig=True)

Create level1match quicklook for a given case.

Parameters

casestr

Date string in format YYYYMMDD

configdict

Configuration dictionary

skipExistingbool, optional

Whether to skip if output exists, by default True

versionstr, optional

Version identifier, by default __version__

plotCompleteOnlybool, optional

Whether to only plot if data is complete, by default True

returnFigbool, optional

Whether to return the figure, by default True

Returns

tuple

(output_file_path, matplotlib.figure.Figure) if returnFig=True, otherwise (output_file_path, None)

VISSSlib.quicklooks.metaRotationYearlyQuicklook(year, config, version='0.0', skipExisting=True)

Create yearly meta rotation quicklook for a given year.

Parameters

yearstr

Year string in format YYYY

configdict or str

Configuration dictionary or path to config file

versionstr, optional

Version identifier, by default __version__

skipExistingbool, optional

Whether to skip if output exists, by default True

Returns

tuple

(output_file_path, matplotlib.figure.Figure)

VISSSlib.quicklooks.metaRotationQuicklook(case, config, version='0.0', skipExisting=True)

Create meta rotation quicklook for a given case.

Parameters

casestr

Date string in format YYYYMMDD

configdict

Configuration dictionary

versionstr, optional

Version identifier, by default __version__

skipExistingbool, optional

Whether to skip if output exists, by default True

Returns

tuple

(output_file_path, matplotlib.figure.Figure)

VISSSlib.quicklooks.createLevel2detectQuicklook(case, camera, config, version='0.0', skipExisting=True, returnFig=True)

Create level2detect quicklook for a given case and camera.

Parameters

casestr

Date string in format YYYYMMDD

camerastr

Camera identifier

configdict

Configuration dictionary

versionstr, optional

Version identifier, by default __version__

skipExistingbool, optional

Whether to skip if output exists, by default True

returnFigbool, optional

Whether to return the figure, by default True

Returns

tuple

(output_file_path, matplotlib.figure.Figure) if returnFig=True, otherwise (output_file_path, None)

VISSSlib.quicklooks.createLevel2matchQuicklook(case, config, version='0.0', skipExisting=True, returnFig=True)

Create level2match quicklook for a given case.

Parameters

casestr

Date string in format YYYYMMDD

configdict

Configuration dictionary

versionstr, optional

Version identifier, by default __version__

skipExistingbool, optional

Whether to skip if output exists, by default True

returnFigbool, optional

Whether to return the figure, by default True

Returns

tuple

(output_file_path, matplotlib.figure.Figure) if returnFig=True, otherwise (output_file_path, None)

VISSSlib.quicklooks.createLevel2trackQuicklook(case, config, version='0.0', skipExisting=True, returnFig=True)

Create level2track quicklook for a given case.

Parameters

casestr

Date string in format YYYYMMDD

configdict

Configuration dictionary

versionstr, optional

Version identifier, by default __version__

skipExistingbool, optional

Whether to skip if output exists, by default True

returnFigbool, optional

Whether to return the figure, by default True

Returns

tuple

(output_file_path, matplotlib.figure.Figure) if returnFig=True, otherwise (output_file_path, None)

VISSSlib.quicklooks.createLevel1matchParticlesQuicklook(timestamp, config, version='0.0', container_width=200, container_height_max=300, nTiles=60, nRows=4, extra=1, readParticlesFromFiles=True, skipExisting=True, ffOut='default', timeStep='variable', minBlur='config', minSize='config', omitLabel4small='config', timedelta=np.timedelta64(1, 'D'), returnFig=True, doLevel1matchQuicklook=True)

Create level1match particles quicklook for a given timestamp.

Parameters

timestampstr

Timestamp string in format YYYYMMDDHH or YYYYMMDD

configdict

Configuration dictionary

versionstr, optional

Version identifier, by default __version__

container_widthint, optional

Width of each tile container, by default 200

container_height_maxint, optional

Maximum height of each tile container, by default 300

nTilesint, optional

Number of tiles per row, by default 60

nRowsint, optional

Number of rows in output, by default 4

extraint, optional

Extra spacing between tiles, by default 1

readParticlesFromFilesbool, optional

Whether to read particles from files, by default True

skipExistingbool, optional

Whether to skip if output exists, by default True

ffOutstr, optional

Output file path, by default “default”

timeStepstr, optional

Time step method (‘fixed’ or ‘variable’), by default “variable”

minBlurfloat or str, optional

Minimum blur threshold, by default “config”

minSizeint or str, optional

Minimum particle size, by default “config”

omitLabel4smallbool or str, optional

Whether to omit labels for small particles, by default “config”

timedeltanumpy.timedelta64, optional

Time window for data selection, by default np.timedelta64(1, “D”)

returnFigbool, optional

Whether to return the figure, by default True

doLevel1matchQuicklookbool, optional

Whether to also run level1match quicklook, by default True

Returns

tuple

(output_file_path, matplotlib.figure.Figure) if returnFig=True, otherwise (output_file_path, None)

VISSSlib.quicklooks.createLevel3RimingQuicklook(case, config, skipExisting=True, version='0.0', returnFig=True)

Create level3 combined rime quicklook for a given case.

Parameters

casestr

Date string in format YYYYMMDD

configdict

Configuration dictionary

skipExistingbool, optional

Whether to skip if output exists, by default True

versionstr, optional

Version identifier, by default __version__

returnFigbool, optional

Whether to return the figure, by default True

Returns

matplotlib.figure.Figure

The generated figure

VISSSlib.tools.loopify_with_camera(func=None, *, endYesterday=True)

Decorator to make function loop over cases and cameras.

Parameters

endYesterdaybool, optional

Whether to end the case range at yesterday. Default is True. This parameter is passed to getCaseRange.

Returns

callable

Decorator function or wrapped function.

Examples

@loopify_with_camera def my_func(case, camera, config):

pass

@loopify_with_camera() def my_func(case, camera, config):

pass

@loopify_with_camera(endYesterday=False) def my_func(case, camera, config):

pass

VISSSlib.tools.loopify(func=None, *, endYesterday=True)

Decorator to make function loop over cases.

Parameters

endYesterdaybool, optional

Whether to end the case range at yesterday. Default is True. This parameter is passed to getCaseRange.

Returns

callable

Decorator function or wrapped function.

Examples

@loopify def my_func(case, config):

pass

@loopify() def my_func(case, config):

pass

@loopify(endYesterday=False) def my_func(case, config):

pass

class VISSSlib.tools.DictNoDefault(*args, **kwargs)

Bases: Dict

Dictionary class that raises KeyError when accessing missing keys.

Inherits from addict.Dict.

VISSSlib.tools.nicerNames(string)

Replace VISSS names with up-to-date equivalents.

Parameters

stringstr

Input string to process.

Returns

str

String with replaced names.

VISSSlib.tools.readSettings(fname)

Read configuration settings from a YAML file. Default is taken from DEFAULT_SETTINGS

Parameters

fnamestr or dict

Path to YAML file or already parsed config dict.

Returns

addict.Dict

Configuration settings.

VISSSlib.tools.getCaseRange(nDays, config, endYesterday=True)

Get list of case identifiers from a date range.

Parameters

nDaysint, str

Number of days going back or date string “YYYYMMDD” or “YYYYMMDD-YYYYMMDD” or “YYYYMMDD,YYYYMMDD,YYYYMMDD”.

configdict

Configuration settings.

endYesterdaybool, optional

Whether to end yesterday, by default True.

Returns

list of str

List of case identifiers.

VISSSlib.tools.getDateRange(nDays, config, endYesterday=True)

Generate date range based on input parameters.

Parameters

nDaysint, str

Number of days going back or date string “YYYYMMDD” or “YYYYMMDD-YYYYMMDD” or “YYYYMMDD,YYYYMMDD,YYYYMMDD”.

configdict

Configuration settings.

endYesterdaybool, optional

Whether to end yesterday, by default True.

Returns

pandas.DatetimeIndex

Date range.

VISSSlib.tools.getPreviousKey(thisDict, case)

Get the previous key-value pair in a dictionary based on timestamp.

Parameters

thisDictdict

Dictionary with datetime keys.

casestr

Case identifier.

Returns

tuple

Previous value and timestamp, or None if not found.

VISSSlib.tools.getPreviousCalibrationOffset(case, config)

Get the previous calibration offset for a given case.

Parameters

casestr

Case identifier.

configdict

Configuration settings.

Returns

tuple

Calibration offset and timestamp, or None if not found.

VISSSlib.tools.getMode(a)

Compute the mode of an array.

Parameters

aarray_like

Input array.

Returns

ndarray

Mode of the array.

VISSSlib.tools.open_mfmetaFrames(fnames, config, start=None, end=None, skipFixes=[])

Open multiple metaFrame files.

Parameters

fnameslist of str

File names.

configdict

Configuration settings.

startdatetime, optional

Start time, by default None.

enddatetime, optional

End time, by default None.

skipFixeslist of str, optional

Fixes to skip, by default [].

Returns

xarray.Dataset

Combined dataset.

VISSSlib.tools.open_mflevel1detect(fnamesExt, config, start=None, end=None, skipFixes=[], datVars='all')

Open multiple level1detect files.

Parameters

fnamesExtlist of str

File names.

configdict

Configuration settings.

startdatetime, optional

Start time, by default None.

enddatetime, optional

End time, by default None.

skipFixeslist of str, optional

Fixes to skip, by default [].

datVarsstr or list of str, optional

Variables to include, by default “all”.

Returns

xarray.Dataset

Combined dataset.

VISSSlib.tools.open_mflevel1match(fnamesExt, config, datVars='all')

Open multiple level1match files.

Parameters

fnamesExtlist of str

File names.

configdict

Configuration settings.

datVarsstr or list of str, optional

Variables to include, by default “all”.

Returns

xarray.Dataset

Combined dataset.

VISSSlib.tools.identifyBlockedBlowingSnowData(fnames, config, timeIndex1, sublevel)

Identify blocked blowing snow data.

Parameters

fnameslist of str

File names.

configdict

Configuration settings.

timeIndex1array_like

Time bins.

sublevelstr

Sublevel identifier.

Returns

tuple of xarray.DataArray

Blowing snow ratio and detected particles.

VISSSlib.tools.compareNDetected(nDetectedL, nDetectedF)

Compare detected particles between two cameras.

Parameters

nDetectedLxarray.DataArray

Detected particles for leader.

nDetectedFxarray.DataArray

Detected particles for follower.

Returns

xarray.DataArray

Ratio of minimum to maximum detections.

VISSSlib.tools.removeBlockedBlowingData(dat1D, events, config, threshold=0.1)

Remove blocked or blowing snow data.

Parameters

dat1Dxarray.Dataset

Input dataset.

eventsstr or xarray.Dataset

Events file or dataset.

configdict

Configuration settings.

thresholdfloat, optional

Blocking threshold, by default 0.1 i.e. 10%

Returns

xarray.Dataset

Filtered dataset or None if empty.

VISSSlib.tools.estimateCaptureIdDiff(leaderFile, followerFiles, config, dim, concat_dim='capture_time', nPoints=500, maxDiffMs=1)

Estimate capture ID difference between cameras.

Parameters

leaderFilestr

Leader camera file.

followerFileslist of str

Follower camera files.

configdict

Configuration settings.

dimstr

Dimension to use.

concat_dimstr, optional

Concatenation dimension, by default “capture_time”.

nPointsint, optional

Number of points to sample, by default 500.

maxDiffMsint, optional

Maximum difference in milliseconds, by default 1.

Returns

tuple

Estimated ID difference and number of samples.

VISSSlib.tools.estimateCaptureIdDiffCore(leaderDat, followerDat, dim, nPoints=500, maxDiffMs=1, timeDim='capture_time')

Core function to estimate capture ID difference.

Parameters

leaderDatxarray.Dataset

Leader data.

followerDatxarray.Dataset

Follower data.

dimstr

Dimension to use.

nPointsint, optional

Number of points to sample, by default 500.

maxDiffMsint, optional

Maximum difference in milliseconds, by default 1.

timeDimstr, optional

Time dimension, by default “capture_time”.

Returns

tuple

Estimated ID difference and number of samples.

Raises

RuntimeError

If capture ID varies too much.

VISSSlib.tools.otherCamera(camera, config)

Get the other VISSS camera based on configuration.

Parameters

camerastr

Camera identifier.

configdict

Configuration settings.

Returns

str

Other camera identifier.

Raises

ValueError

If camera is not in instruments list.

VISSSlib.tools.cutFollowerToLeader(leader, follower, gracePeriod=1, dim='fpid')

Cut follower data to match leader data with respect to time.

Parameters

leaderxarray.Dataset

Leader data.

followerxarray.Dataset

Follower data.

gracePeriodint, optional

Grace period in seconds, by default 1.

dimstr, optional

Dimension to use, by default “fpid”.

Returns

xarray.Dataset

Cut follower data.

VISSSlib.tools.nextCase(case)

Get the next case identifier.

Parameters

casestr

Current case identifier.

Returns

str

Next case identifier.

VISSSlib.tools.prevCase(case)

Get the previous case identifier.

Parameters

casestr

Current case identifier.

Returns

str

Previous case identifier.

VISSSlib.tools.timestamp2case(dd)

Convert timestamp to case identifier.

Parameters

dddatetime

Timestamp.

Returns

str

Case identifier.

VISSSlib.tools.case2timestamp(case)

Convert case identifier to timestamp.

Parameters

casestr

Case identifier.

Returns

numpy.datetime64

Timestamp.

Raises

NotImplementedError

If long cases are not implemented.

VISSSlib.tools.rescaleImage(frame1, rescale, anti_aliasing=False, anti_aliasing_sigma=None, mode='edge')

Rescale an image.

Parameters

frame1array_like

Input image.

rescalefloat

Scaling factor.

anti_aliasingbool, optional

Apply anti-aliasing, by default False.

anti_aliasing_sigmafloat, optional

Anti-aliasing sigma, by default None.

modestr, optional

Resizing mode, by default “edge”.

Returns

array_like

Rescaled image.

VISSSlib.tools.displayImage(frame, doDisplay=True, rescale=None)

Display an image.

Parameters

framearray_like

Input image.

doDisplaybool, optional

Whether to display, by default True.

rescalefloat, optional

Scaling factor, by default None.

Returns

IPython.display.Image or None

Image object if doDisplay=False, otherwise None.

class VISSSlib.tools.ZipFile(file, **kwargs)

Bases: ZipFile

Extended zip file class with automatic parent directory creation. Extra functions to store and retrieve images as numpy arrays

addnpy(fname, array)

extractnpy(fname)

VISSSlib.tools.imageZipFile(fname, **kwargs)

Create appropriate archive file handler.

Parameters

fnamestr

File name.

**kwargsdict

Additional arguments for archive creation.

Returns

Archive handler

ZipFile or BlockImageArchive instance.

class VISSSlib.tools.BlockImageArchive(filepath, mode='r', block_size=256, level=8)

Bases: object

A high-efficiency binary archive for storing thousands of small numpy arrays.

This class provides a single-file storage solution specifically optimized for small, variable-sized uint8 arrays. It uses block compression to maximize the compression ratio and maintains a JSON index for O(1) random access.

Parameters

filepathstr

Path to the archive file.

mode{‘a’, ‘w’, ‘r’}, optional

‘a’ : Append/Read mode. Loads existing index and allows adding more data. ‘w’ : Write mode. Overwrites any existing file. ‘r’ : Read-only mode. Prevents any modifications to the file. Default is ‘a’.

block_sizeint, optional

Number of images to group into a single compressed block. Default is 256.

levelint, optional

Zip compression level. Default is 8.

addnpy(image_id, array)

Add a numpy array to the archive.

The array is added to an internal buffer. When the buffer reaches block_size, the images are compressed and written to disk.

Parameters

image_idstr or int

Unique identifier for the image.

arraynumpy.ndarray

The uint8 array to store.

extractnpy(image_id)

Retrieve a numpy array by its ID.

This method reads only the required compressed block from disk, decompresses it in memory, and returns the specific slice.

Parameters

image_idstr or int

The identifier of the image to retrieve.

Returns

numpy.ndarray

The reconstructed uint8 array.

Raises

KeyError

If the image_id is not found in the archive index.

close()

Closes the file handle. In ‘w’ or ‘a’ mode, finalizes the index. In ‘r’ mode, simply closes the file.

VISSSlib.tools.createParentDir(file, mode=None)

Create parent directory if it doesn’t exist.

Parameters

filestr

File path.

modeint, optional

Directory mode, by default None.

VISSSlib.tools.savefig(fig, config, filename, fnames=None, addLogo=True, w_pad=None, h_pad=None, **kwargs)

Save a matplotlib Figure to filename with proper permissions.

This function saves a matplotlib figure to a file with appropriate directory creation and file permissions. It also adds status text to the figure and optionally includes a logo.

Parameters

figmatplotlib.figure.Figure

The matplotlib figure to save.

configdict

Configuration settings containing directory modes and logo information.

filenamestr

The output file path where the figure will be saved.

fnamesstr or list of str, optional

File names used to determine creation date for status text. If None, no date information is included. Default is None.

addLogobool, optional

Whether to add a logo to the figure. Default is True.

w_padfloat, optional

Width padding for tight layout. Default is None.

h_padfloat, optional

Height padding for tight layout. Default is None.

**kwargsdict

Additional keyword arguments passed to matplotlib’s savefig function.

Returns

matplotlib.figure.Figure

The figure that was saved (for chaining operations).

Notes

• Creates parent directories if they don’t exist with permissions based on config.dirMode

• Sets file permissions to config.fileMode after saving

• Adds status text with VISSSlib version, creation timestamp, and file creation date

• Optionally adds a logo from config.logo if available

• Removes existing files with the same name before saving to prevent corruption

• Uses tight_layout with specified padding for better figure formatting

Example

>>> import matplotlib.pyplot as plt
>>> fig, ax = plt.subplots()
>>> ax.plot([1, 2, 3], [1, 4, 9])
>>> savefig(fig, config, "output.png")

VISSSlib.tools.ncAttrs(site, visssGen, extra={})

Generate NetCDF attributes.

Parameters

sitestr

Site identifier.

visssGenstr

Generator information.

extradict, optional

Extra attributes, by default {}.

Returns

dict

NetCDF attributes.

VISSSlib.tools.finishNc(dat, site, visssGen, extra={})

Finalize NetCDF dataset with attributes and encoding.

Parameters

datxarray.Dataset

Dataset to finalize.

sitestr

Site identifier.

visssGenstr

Generator information.

extradict, optional

Extra attributes, by default {}.

Returns

xarray.Dataset

Finalized dataset.

VISSSlib.tools.getPrevRotationEstimates(datetime64, config)

Extract rotation estimates from config.

Parameters

datetime64numpy.datetime64

Datetime to query.

configdict

Configuration settings.

Returns

tuple

Rotation matrix, error matrix, and timestamp.

VISSSlib.tools.getPrevRotationEstimate(datetime64, key, config)

Extract single rotation estimate from config.

Parameters

datetime64numpy.datetime64

Datetime to query.

keystr

Key for rotation estimate.

configdict

Configuration settings.

Returns

tuple

Rotation estimate and timestamp.

Raises

RuntimeError

If no rotation estimate found.

VISSSlib.tools.rotXr2dict(dat, config=None)

Convert rotation xarray data to dictionary.

Parameters

datxarray.Dataset

Rotation data.

configdict, optional

Configuration settings, by default None.

Returns

dict

Configuration dictionary with rotation info.

VISSSlib.tools.rotDict2Xr(rotate, rotate_err, prevTime)

Convert rotation dictionary to xarray dataset.

Parameters

rotatedict

Rotation matrix.

rotate_errdict

Rotation error matrix.

prevTimenumpy.datetime64

Previous timestamp.

Returns

xarray.Dataset

Rotation dataset.

VISSSlib.tools.execute_stdout(command)

Execute command and print output continuously.

Parameters

commandstr

Command to execute.

Returns

tuple

Exit code and output.

VISSSlib.tools.concat(*strs)

Concatenate strings with spaces.

Parameters

*strsstr

Strings to concatenate.

Returns

str

Concatenated string.

VISSSlib.tools.concatImgY(im1, im2, background=0)

Concatenate images vertically.

Parameters

im1array_like

Upper image.

im2array_like

Lower image.

backgroundint, optional

Background color, by default 0.

Returns

array_like

Concatenated image.

VISSSlib.tools.concatImgX(im1, im2, background=0)

Concatenate images horizontally.

Parameters

im1array_like

Left image.

im2array_like

Right image.

backgroundint, optional

Background color, by default 0.

Returns

array_like

Concatenated image.

VISSSlib.tools.open2(file, config, mode='r', cleanUp=True, **kwargs)

Open file with directory creation and permissions.

Parameters

filestr

File path.

configdict

Configuration settings.

modestr, optional

File mode, by default “r”.

cleanUpbool, optional

Clean up temporary files, by default True.

**kwargsdict

Additional arguments for opening.

Returns

file handle

Opened file handle.

VISSSlib.tools.tryRemovingFile(file)

Try to remove a file.

Parameters

filestr

File path.

VISSSlib.tools.to_netcdf2(dat, config, file, **kwargs)

Save dataset to NetCDF with directory creation. Write to random file and move to final file to avoid errors due to race conditions or exisiting files

Parameters

datxarray.Dataset

Dataset to save.

configdict

Configuration settings.

filestr

Output file name.

**kwargsdict

Additional arguments for saving.

Returns

None

VISSSlib.tools.linreg(x, y)

Perform linear regression. Turns out to be a lot faster than scipy and numpy code when using numba (1.7 us vs 25 us)

Parameters

xarray_like

Independent variable.

yarray_like

Dependent variable.

Returns

tuple

Slope and intercept of regression line.

VISSSlib.tools.cart2pol(x, y)

Convert Cartesian coordinates to polar.

Parameters

xarray_like

X coordinates.

yarray_like

Y coordinates.

Returns

tuple

Polar coordinates (rho, phi).

VISSSlib.tools.runCommandInQueue(IN, stdout=-3)

Run command in queue with file locking.

Parameters

INtuple

Command and output file.

stdoutfile handle, optional

Standard output, by default subprocess.DEVNULL.

Returns

bool

Success status.

VISSSlib.tools.worker1(queue, ww=0, status=None, waitTime=5)

Worker function for processing queue items.

Parameters

queuestr

Queue name.

wwint, optional

Worker number, by default 0.

statusarray, optional

Status array, by default None.

waitTimeint, optional

Wait time between checks, by default 5.

Returns

None

VISSSlib.tools.workers(queue, nJobs=2, waitTime=60, join=True)

Start multiple worker processes.

Parameters

queuestr

Queue name.

nJobsint, optional

Number of jobs, by default os.cpu_count().

waitTimeint, optional

Wait time between checks, by default 60.

joinbool, optional

Join processes, by default True.

Returns

list

Worker processes.

VISSSlib.tools.copyCurrentQuicklook(level, ff)

Copy the latest quicklook file to the current quicklook location.

This function copies the most recently generated quicklook file for a given processing level to a corresponding “current” location. This is typically used to maintain a symlink or copy of the most recent quicklook image for easy access.

Parameters

levelstr

Processing level for which to copy the quicklook (e.g., ‘level1detect’).

ffobject

File finder object containing file information and quicklook paths.

Returns

None

This function does not return a value but performs file copying operations.

Notes

The function only copies the quicklook file if the date matches today’s date. This prevents overwriting previous quicklook files with potentially outdated images from previous days.

VISSSlib.tools.checkForExisting(ffOut, level0=None, events=None, parents=None)

Check if file exists and is up-to-date including potential parents.

Parameters

ffOutstr

Output file path.

level0list, optional

Level 0 data files, by default None.

eventslist, optional

Event files, by default None.

parentslist, optional

Parent files, by default None.

Returns

bool

True if file should be regenerated.

VISSSlib.tools.unpackQualityFlags(quality, doubleTimestamps=False)

Unpack quality flags into boolean array.

Parameters

qualityxarray.DataArray

Quality flags.

doubleTimestampsbool, optional

Double timestamps for plotting, by default False.

Returns

xarray.DataArray

Expanded quality flags.

VISSSlib.tools.timestamp2str(ts)

Convert timestamp to string.

Parameters

tsint

Timestamp.

Returns

str

Formatted timestamp string.

VISSSlib.tools.copyLastMetaRotation(config, fromCase, toCase)

Copy the last meta rotation file from one case to another.

This function copies the most recent meta rotation file from a source case to a destination case, updating the timestamp in the filename accordingly.

Parameters

configdict

Configuration settings.

fromCasestr

Source case identifier (e.g., “20230101”).

toCasestr

Destination case identifier (e.g., “20230102”).

Returns

xarray.Dataset or None

The copied meta rotation dataset or None if no rotation file exists.

Notes

This function is decorated with @loopify, which means it will automatically iterate over all cases if a range is provided. For example, if fromCase is “20230101-20230103”, it will process all three cases.

VISSSlib.tools.reportLastFiles(settings, writeFile=True, nameFile=False, products=['level0txt', 'level0', 'metaFrames', 'level1detect', 'metaRotation', 'level1match', 'level1track', 'level2match', 'level2track'])

report last available files for various processing levels

Parameters

settingsstr

VISSS settings YAML file

writeFilebool, optional

write output to file (the default is True)

nameFilebool, optional

name last files (the default is False)

productslist, optional

list of products to be summarized (the default is [ “level0txt”, “level0”, “metaFrames”, “level1detect”, “metaRotation”, “level1match”, “level1track”, “level2match”, “level2track”, ])

VISSSlib.tracking.myKF(FirstPos3D, velocityGuess=[0, 0, 50], R_std=2, q_var=1)

Initialize a Kalman Filter for 3D particle tracking.

Parameters

FirstPos3Darray_like

Initial 3D position [x, y, z] of the particle.

velocityGuessarray_like, optional

Initial velocity guess [vx, vy, vz], default [0, 0, 50].

R_stdfloat, optional

Standard deviation for measurement noise, default 2.

q_varfloat, optional

Variance for process noise, default 1.

Returns

kffilterpy.kalman.KalmanFilter

Configured Kalman Filter instance.

Notes

The filter uses a constant velocity model with state vector [x, vx, y, vy, z, vz].

class VISSSlib.tracking.Track(position, feature, size, trackIdCount, startTime, velocityGuess=[0, 0, 50], R_std=2, q_var=1, reduced_q_var=0.5)

Bases: object

Track class for every object to be tracked

property lastAngle

Calculate the angle of the last movement vector relative to vertical.

Returns

float

Angle in radians between the last movement vector and vertical axis. Returns np.nan if there’s not enough data.

property meanVelocity

Calculate the mean velocity over the track history.

Returns

array_like

Mean velocity vector [vx, vy, vz].

property length

real lenght without nans

property trace

Get the track’s position history.

Returns

array_like

Array of 3D positions [x, y, z] over time.

property meanSize

Calculate the mean size over the track history.

Returns

float

Mean particle size.

updateTrack(position, feature, size)

Update track with new observation.

Parameters

positionarray_like or None

New 3D position [x, y, z]. If None, uses prediction.

featurearray_like

New feature vector.

sizefloat

New particle size.

Notes

When position is None, the track is updated with NaN positions and the last known feature is reused.

predict()

Predict next state using Kalman Filter.

Returns

predictedPosarray_like

Predicted 3D position [x, y, z].

predictedVelarray_like

Predicted velocity [vx, vy, vz].

predictedAngfloat

Predicted angle from vertical (radians).

class VISSSlib.tracking.Tracker(lv1match, config, dist_thresh=4, max_trace_length=None, velocityGuessXY=[0, 0], maxIter=1e+30, fig=None, featureVariance={'Dmax': 1, 'distance': 40000}, minTrackLen4training=4, maxAge4training=300, costExperiencePenalty=array([1, 1, 6, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]), velSlope=None, velIntercept=None, R_std=2, q_var=1, reduced_q_var=1, training=True, verbosity=0)

Bases: object

Tracker class that updates track vectors of object tracked

updateAll()

Process all frames for tracking.

Returns

lv1trackxarray.Dataset

Tracked particle data with track IDs.

velGuess_slopefloat

Slope of size-velocity relationship.

velGuess_interceptfloat

Intercept of size-velocity relationship.

Notes

Runs in two passes if in training mode: 1. Training pass: learns size-velocity relationship 2. Tracking pass: applies learned model to entire dataset

property activeTrackLength

Get lengths of all active tracks.

Returns

array_like

Array containing the length of each active track.

update(ff)

Process one frame of particle detections.

Parameters

ffint

Frame index.

Notes

Steps: 1. Extract particle positions and features 2. Predict next state for active tracks 3. Calculate assignment cost matrix 4. Apply Hungarian algorithm for assignment 5. Handle unassigned detections (new tracks) 6. Update tracks with new measurements 7. Remove stale tracks

getFeatures(features, pair_id)

Get features for a specific particle.

Parameters

featuresarray_like or None

Feature array for all particles.

pair_idint

Index of the particle to get features for.

Returns

tuple

Tuple of (feature_vector, size, velocity_guess)

save(pair_id, track)

save results

updateVelocityFirstGuess(capture_times, ff)

Update velocity first guess using recent track data.

Parameters

capture_timesarray_like

Current frame capture times.

ffint

Frame index.

Notes

Fits a new size-velocity model when sufficient recent data exists. Resets to defaults when data is stale or insufficient.

getVelocityFirstGuess(size)

Estimate initial velocity from particle size.

Parameters

sizefloat

Particle size.

Returns

velocityGuesslist

Velocity vector [vx, vy, vz] in mm/s.

Notes

Uses logarithmic relationship: log10(v) = slope*log10(size) + intercept

reset()

Reset active tracks and archive recent track data.

Notes

Moves active tracks to archive and clears active track list. Maintains only recent archive data (last backSteps*10 items).

removeTracks(del_ii)

Remove inactive tracks and archive their data.

Parameters

del_iiarray_like

Indices of tracks to remove.

Notes

Archives track data before removing from active list. Maintains only recent archive data (last backSteps*10 items).

VISSSlib.tracking.trackParticles(fnameLv1Detect, config, version='0.0', dist_thresh=4, fig=None, max_trace_length=None, velocityGuessXY=[0, 0], maxIter=1e+30, featureVariance={'Dmax': 1, 'distance': 40000}, minMatchScore=0.001, minTrackLen4training=2, maxAge4training=100, costExperiencePenalty=array([1, 1, 6, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]), R_std=2, q_var=1, reduced_q_var=1, doMatchIfRequired=False, writeNc=True, skipExisting=True, verbosity=0)

Main particle tracking workflow.

Parameters

fnameLv1Detectstr

Level1 detection filename.

configobject

Configuration settings.

versionstr, optional

Processing version, default __version__.

dist_threshfloat, optional

Assignment distance threshold, default 4.

figmatplotlib.figure.Figure, optional

Figure for visualization, default None.

max_trace_lengthint, optional

Maximum track history length, default None.

velocityGuessXYlist, optional

Initial XY velocity guess [vx, vy], default [0,0].

maxIterint, optional

Maximum frames to process, default 1e30.

featureVariancedict, optional

Feature variances for cost calculation, default {“distance”:40000, “Dmax”:1}.

minMatchScorefloat, optional

Minimum match score for particles, default 1e-3.

minTrackLen4trainingint, optional

Minimum track length for training, default 2.

maxAge4trainingfloat, optional

Maximum training data age (seconds), default 100.

costExperiencePenaltyarray_like, optional

Assignment penalty factors by track length.

R_stdfloat, optional

Measurement noise standard deviation, default 2.

q_varfloat, optional

Process noise variance, default 1.

reduced_q_varfloat, optional

Reduced process noise after first step, default 1.

doMatchIfRequiredbool, optional

Run matching if missing, default False.

writeNcbool, optional

Write netCDF output, default True.

skipExistingbool, optional

Skip processing if output exists, default True.

verbosityint, optional

Verbosity level, default 0.

Returns

lv1trackxarray.Dataset or None

Tracked particle data, None if skipped/broken.

fnameTrackingstr

Output filename.

Notes

Handles data loading, particle matching (if needed), and tracking. Creates output files and handles existing/skipped cases.