binaryFileStorage
Class BinaryStore

java.lang.Object
  PamController.PamControlledUnit
      binaryFileStorage.BinaryStore

All Implemented Interfaces:

OfflineDataStore, PamSettings, PamSettingsSource

public class BinaryStore
extends PamControlledUnit
implements PamSettings, PamSettingsSource, OfflineDataStore

The binary store will work very much like the database in that it monitors the output of data streams and when data is added to them it writes it to the binary store.

The binary store has a number of advantages over database storage, particularly when it comes to writing objects with varying record lengths such as clicks and whistle contours.

Further information about binary storage and information on formats can be found here.

Author:

Doug Gillespie

Nested Class Summary

(package private) class	BinaryStore.BinaryDataMapMaker
(package private) class	BinaryStore.BinaryStorageOptions
(package private) class	BinaryStore.NewFileTask
(package private) class	BinaryStore.StoredUnitServer Class to serve up the next data unit that needs to be stored from a particular file.

Field Summary

private BinaryMapMakingDialog	binaryMapDialog
private BinarySettingsStorage	binarySettingsStorage
private BinaryStoreProcess	binaryStoreProcess
protected BinaryStoreSettings	binaryStoreSettings
static int	CURRENT_FORMAT
static java.lang.String	fileType
static java.lang.String	indexFileType
private PamController	pamController
static java.lang.String	settingsFileType
private java.util.Vector<BinaryOutputStream>	storageStreams
static java.lang.String	unitType

Fields inherited from class PamController.PamControlledUnit

isViewer

Constructor Summary

BinaryStore(java.lang.String unitName)

Method Summary

private boolean	checkMemory(PamDataBlock dataBlock) Check available memory.
private boolean	checkMemory(PamDataBlock dataBlock, long minAmount)
private void	closeStores()
private void	createBinaryDataMaps(java.util.ArrayList<PamDataBlock> streams)
private java.util.ArrayList<java.io.File>	createChangedFileList(PamDataBlock pamDataBlock) Create a list of files which have data in that datablock that has been flagged as changed.
javax.swing.JMenuItem	createFileMenu(javax.swing.JFrame parentFrame)
boolean	createMapPoint(java.io.File aFile, BinaryHeaderAndFooter bhf, java.util.ArrayList<PamDataBlock> streams) Create a data map point and add it to the map.
private void	createNewStores()
void	createOfflineDataMap(java.awt.Window parentFrame) Create a basic map of the data including first and last times and some kind of data/unit time count plus ideally some kind of start and stop time list of where there are gaps in the data.
private void	doInitialStoreChecks()
private PamDataBlock	findDataStream(BinaryHeader binaryHeader, java.util.ArrayList<PamDataBlock> streams) Find the datastream to go with a particular file header.
java.io.File	findIndexFile(java.io.File dataFile, boolean checkExists) Find the index file to match a given data file.
private BinaryOfflineDataMapPoint	findMapPoint(PamDataBlock pamDataBlock, java.io.File file) Find a data map point for a specific file.
BinaryStoreSettings	getBinaryStoreSettings()
java.lang.String	getDataSourceName() Get the data source name
BinaryHeaderAndFooter	getFileHeaderAndFooter(java.io.File file) Run checks on each file.
java.lang.String	getFolderName(long timeStamp, boolean addSeparator) Get the name of the current storage folder and check the folder exists, if necessary, creating it.
int	getNumSettings() Get the number of different settings within the settings source.
PamSettingsGroup	getSettings(int settingsIndex) Get a specific PamSettingsGroup
java.io.Serializable	getSettingsReference()
java.lang.String	getSettingsSourceName() Get a name for the settings source
long	getSettingsVersion()
private java.util.ArrayList<PamDataBlock>	getStreamingDataBlocks()
java.lang.String	getUnitName() Returns the name of the unit
java.lang.String	getUnitType()
java.util.List<java.io.File>	listAllFiles() List all the files in the binary storage folder
private void	listDataFiles(java.util.ArrayList<java.io.File> fileList, java.io.File folder, PamFileFilter filter) List all data files - get's called recursively
boolean	loadData(PamDataBlock dataBlock, BinaryOfflineDataMapPoint mapPoint, long dataStart, long dataEnd) Load the data from a single file.
boolean	loadData(PamDataBlock dataBlock, long dataStart, long dataEnd) Load data for a given datablock between two time limits.
private void	makeInitialDataMap()
void	notifyModelChanged(int changeType) General notification when the PAMGAURD model changes.
private void	openStores()
void	pamHasStopped() Called for all controlled units after Pam acquisition has stopped
void	pamToStart() called just before data acquisition starts.
private void	prepareStores()
private BinaryHeaderAndFooter	readHeadAndFoot(java.io.File file) Read the header and footer from a binary file.
protected void	reOpenStores(int endReason) Called from the process to close and reopen each datastream in a new file.
private boolean	reportError(java.lang.String string)
boolean	restoreSettings(PamControlledUnitSettings pamControlledUnitSettings)
private boolean	saveData(java.io.File file, PamDataBlock pamDataBlock) Rewrite a specific file using data from pamDataBlock.
boolean	saveData(PamDataBlock pamDataBlock) Saves data in a PamDataBlock.
boolean	saveStartSettings(long timeNow) Save the settings in some way or another.
void	setBinaryStoreSettings(BinaryStoreSettings binaryStoreSettings)
private void	sortBinaryDataMaps(java.util.ArrayList<PamDataBlock> streams)

Methods inherited from class PamController.PamControlledUnit

addOtherRelatedMenuItems, addPamProcess, addRelatedMenuItems, canClose, createDetectionMenu, createDisplayMenu, createHelpMenu, fillXMLElement, fillXMLParameters, getFrameNumber, getNumPamProcesses, getPamController, getPamModuleInfo, getPamProcess, getPamView, getSidePanel, getTabClipCopier, getTabPanel, getTabSpecificMenuBar, gotoTab, notifyArrayChanged, removePamProcess, removeUnit, rename, saveViewerData, setFrameNumber, setPamController, setPamModuleInfo, setPamView, setSidePanel, setTabPanel, setupControlledUnit, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

fileType

public static final java.lang.String fileType

See Also:

Constant Field Values

indexFileType

public static final java.lang.String indexFileType

See Also:

Constant Field Values

settingsFileType

public static final java.lang.String settingsFileType

See Also:

Constant Field Values

CURRENT_FORMAT

public static int CURRENT_FORMAT

pamController

private PamController pamController

binaryStoreSettings

protected BinaryStoreSettings binaryStoreSettings

storageStreams

private java.util.Vector<BinaryOutputStream> storageStreams

binaryStoreProcess

private BinaryStoreProcess binaryStoreProcess

binarySettingsStorage

private BinarySettingsStorage binarySettingsStorage

unitType

public static final java.lang.String unitType

See Also:

Constant Field Values

binaryMapDialog

private BinaryMapMakingDialog binaryMapDialog

Constructor Detail

BinaryStore

public BinaryStore(java.lang.String unitName)

Method Detail

notifyModelChanged

public void notifyModelChanged(int changeType)

Description copied from class: PamControlledUnit

General notification when the PAMGAURD model changes.

Overrides:

notifyModelChanged in class PamControlledUnit

Parameters:

changeType - type of change

pamToStart

public void pamToStart()

Description copied from class: PamControlledUnit

called just before data acquisition starts. Note that PamObservers get a call to setSampleRate anyway so this mainly needs to be used for display elements that may need their scales adjusted before startup.

Overrides:

pamToStart in class PamControlledUnit

pamHasStopped

public void pamHasStopped()

Description copied from class: PamControlledUnit

Called for all controlled units after Pam acquisition has stopped

Overrides:

pamHasStopped in class PamControlledUnit

prepareStores

private void prepareStores()

reOpenStores

protected void reOpenStores(int endReason)

Called from the process to close and reopen each datastream in a new file. Probably gets called about once an hour on the hour.

openStores

private void openStores()

closeStores

private void closeStores()

createNewStores

private void createNewStores()

getStreamingDataBlocks

private java.util.ArrayList<PamDataBlock> getStreamingDataBlocks()

getSettingsReference

public java.io.Serializable getSettingsReference()

Specified by:

getSettingsReference in interface PamSettings

Returns:

The serialisable object that will be stored

getSettingsVersion

public long getSettingsVersion()

Specified by:

getSettingsVersion in interface PamSettings

Returns:

An integer version number for the settings

getUnitName

public java.lang.String getUnitName()

Description copied from class: PamControlledUnit

Returns the name of the unit

Specified by:

getUnitName in interface PamSettings

Overrides:

getUnitName in class PamControlledUnit

Returns:

the name of the unit

getUnitType

public java.lang.String getUnitType()

Specified by:

getUnitType in interface PamSettings

Overrides:

getUnitType in class PamControlledUnit

Returns:

A Name specific to the type, e.g. Glick detector

getFolderName

public java.lang.String getFolderName(long timeStamp,
                                      boolean addSeparator)

Get the name of the current storage folder and check the folder exists, if necessary, creating it.

Parameters:

timeStamp - current time in milliseconds.

addSeparator - if true, add a path separator character to the end of the path.

Returns:

true if folder exists (or has been created)

restoreSettings

public boolean restoreSettings(PamControlledUnitSettings pamControlledUnitSettings)

Specified by:

restoreSettings in interface PamSettings

Returns:

true if successful The object performs final checks (if needed) and then casts the settings data pamcontrolledunitSettings.settings into the correct type and uses as required

createFileMenu

public javax.swing.JMenuItem createFileMenu(javax.swing.JFrame parentFrame)

Overrides:

createFileMenu in class PamControlledUnit

Parameters:

parentFrame - parent frame for the menu

Returns:

the file menu item

saveStartSettings

public boolean saveStartSettings(long timeNow)

Description copied from interface: PamSettingsSource

Save the settings in some way or another.

Specified by:

saveStartSettings in interface PamSettingsSource

Parameters:

timeNow - current time.

Returns:

true if settings saved successfully

getNumSettings

public int getNumSettings()

Description copied from interface: PamSettingsSource

Get the number of different settings within the settings source.

Specified by:

getNumSettings in interface PamSettingsSource

Returns:

the number of PamSettingsGroups.

getSettings

public PamSettingsGroup getSettings(int settingsIndex)

Description copied from interface: PamSettingsSource

Get a specific PamSettingsGroup

Specified by:

getSettings in interface PamSettingsSource

Parameters:

settingsIndex - index of group

Returns:

a settings group

getSettingsSourceName

public java.lang.String getSettingsSourceName()

Description copied from interface: PamSettingsSource

Get a name for the settings source

Specified by:

getSettingsSourceName in interface PamSettingsSource

Returns:

a name

getBinaryStoreSettings

public BinaryStoreSettings getBinaryStoreSettings()

Returns:

the binaryStoreSettings

setBinaryStoreSettings

public void setBinaryStoreSettings(BinaryStoreSettings binaryStoreSettings)

Parameters:

binaryStoreSettings - the binaryStoreSettings to set

doInitialStoreChecks

private void doInitialStoreChecks()

makeInitialDataMap

private void makeInitialDataMap()

createOfflineDataMap

public void createOfflineDataMap(java.awt.Window parentFrame)

Description copied from interface: OfflineDataStore

Create a basic map of the data including first and last times and some kind of data/unit time count plus ideally some kind of start and stop time list of where there are gaps in the data.

Specified by:

createOfflineDataMap in interface OfflineDataStore

createBinaryDataMaps

private void createBinaryDataMaps(java.util.ArrayList<PamDataBlock> streams)

sortBinaryDataMaps

private void sortBinaryDataMaps(java.util.ArrayList<PamDataBlock> streams)

listAllFiles

public java.util.List<java.io.File> listAllFiles()

List all the files in the binary storage folder

Returns:

List of files.

getFileHeaderAndFooter

public BinaryHeaderAndFooter getFileHeaderAndFooter(java.io.File file)

Run checks on each file.

Parameters:

file - file to check.

createMapPoint

public boolean createMapPoint(java.io.File aFile,
                              BinaryHeaderAndFooter bhf,
                              java.util.ArrayList<PamDataBlock> streams)

Create a data map point and add it to the map.

First the correct data stream (PamDataBlock) must be located then the correct map within that stream. All this searching is necessary since the files will not be read in order so we've no idea up until now which stream it's going to be.

Parameters:

bhf - binary header and footer.

streams - list of data streams

Returns:

true if stream and map found and map point added.

findDataStream

private PamDataBlock findDataStream(BinaryHeader binaryHeader,
                                    java.util.ArrayList<PamDataBlock> streams)

Find the datastream to go with a particular file header.

Parameters:

binaryHeader - binary header read in from a file

streams - Array list of available data streams

Returns:

single stream which matches the header.

readHeadAndFoot

private BinaryHeaderAndFooter readHeadAndFoot(java.io.File file)

Read the header and footer from a binary file.

Parameters:

file -

Returns:

header and footer packed up together.

findIndexFile

public java.io.File findIndexFile(java.io.File dataFile,
                                  boolean checkExists)

Find the index file to match a given data file.

Parameters:

dataFile - data file.

checkExists - check teh file exists and if it doens't return null

Returns:

index file to go with the data file.

listDataFiles

private void listDataFiles(java.util.ArrayList<java.io.File> fileList,
                           java.io.File folder,
                           PamFileFilter filter)

List all data files - get's called recursively

Parameters:

fileList - current fiel list - get's added to

folder - folder to search

filter - file filter

getDataSourceName

public java.lang.String getDataSourceName()

Description copied from interface: OfflineDataStore

Get the data source name

Specified by:

getDataSourceName in interface OfflineDataStore

Returns:

data source name

loadData

public boolean loadData(PamDataBlock dataBlock,
                        long dataStart,
                        long dataEnd)

Description copied from interface: OfflineDataStore

Load data for a given datablock between two time limits.

Specified by:

loadData in interface OfflineDataStore

Parameters:

dataBlock - datablock owner of the data

dataStart - start time in milliseconds

dataEnd - end time in milliseconds

Returns:

true if load successful.

loadData

public boolean loadData(PamDataBlock dataBlock,
                        BinaryOfflineDataMapPoint mapPoint,
                        long dataStart,
                        long dataEnd)

Load the data from a single file.

Generally, PAMGUARD will use the above function which loads between two times, but for some offline analysis tasks, it's convenient to scroll through file at a time in which case this function can be used.

Parameters:

dataBlock - PamDataBlock to receive the data

binaryOfflineDataMapPoint - data map point

dataStart - data start

dataEnd - data end

saveData

public boolean saveData(PamDataBlock pamDataBlock)

Saves data in a PamDataBlock.

First scans all data in the pamDataBlock and works out which files actually need updating based on info in their DataUnitFileInformation then rewrites those files.

Note that data from many files may be in memory and it's also possible that files are only partially read in, in which case, it will be necessary to partially take data from the old file and partially from the stuff in memory !

Specified by:

saveData in interface OfflineDataStore

Parameters:

pamDataBlock - data block holding the data.

Returns:

true if saved Ok.

saveData

private boolean saveData(java.io.File file,
                         PamDataBlock pamDataBlock)

Rewrite a specific file using data from pamDataBlock.

N.B. Not all data from the original file will have been read in, so it will be necessary to take data both from the original file and from the PamDataBlock

Parameters:

file - file to recreate

pamDataBlock -

Returns:

true if file recreated OK, false otherwise.

findMapPoint

private BinaryOfflineDataMapPoint findMapPoint(PamDataBlock pamDataBlock,
                                               java.io.File file)

Find a data map point for a specific file.

reportError

private boolean reportError(java.lang.String string)

createChangedFileList

private java.util.ArrayList<java.io.File> createChangedFileList(PamDataBlock pamDataBlock)

Create a list of files which have data in that datablock that has been flagged as changed.

Parameters:

pamDataBlock -

Returns:

ArrayList of files that have changed data

checkMemory

private boolean checkMemory(PamDataBlock dataBlock)

Check available memory. If it's less than 10 megabytes, return false since we're about to run out.

Returns:

true if there is > 1o Meg of memory left.

checkMemory

private boolean checkMemory(PamDataBlock dataBlock,
                            long minAmount)