Package binaryFileStorage

Class BinaryStore

java.lang.Object
PamController.PamControlledUnit
binaryFileStorage.BinaryStore
All Implemented Interfaces:
DataOutputStore, OfflineDataStore, PamSettings, PamSettingsSource, SettingsNameProvider
Direct Known Subclasses:
SecondaryBinaryStore
public class BinaryStore extends PamControlledUnit implements PamSettings, PamSettingsSource, DataOutputStore
The binary store will work very much like the database in that it monitors the output of data streams and when data is added to them it writes it to the binary store.

The binary store has a number of advantages over database storage, particularly when it comes to writing objects with varying record lengths such as clicks and whistle contours.

Further information about binary storage and information on formats can be found here.

Author:
Doug Gillespie

  • Field Details

  • Constructor Details

    • BinaryStore

      protected BinaryStore(String unitType, String unitName)
      Parameters:
      unitType -
      unitName -

    • BinaryStore

      public BinaryStore(String unitName)

  • Method Details

    • getCurrentFileFormat

      public static int getCurrentFileFormat()

    • getMaxUnderstandableFormat

      public static int getMaxUnderstandableFormat()
      Current maximum understandable format.
      Returns:
      maximum understandable file format.

    • notifyModelChanged

      public void notifyModelChanged(int changeType)
      Description copied from class: PamControlledUnit
      General notification when the PAMGAURD model changes.
      Overrides:
      notifyModelChanged in class PamControlledUnit
      Parameters:
      changeType - type of change

    • pamToStart

      public void pamToStart()
      Description copied from class: PamControlledUnit
      called just before data acquisition starts. Note that PamObservers get a call to setSampleRate anyway so this mainly needs to be used for display elements that may need their scales adjusted before startup.
      Overrides:
      pamToStart in class PamControlledUnit

    • pamHasStopped

      public void pamHasStopped()
      Description copied from class: PamControlledUnit
      Called for all controlled units after Pam acquisition has stopped
      Overrides:
      pamHasStopped in class PamControlledUnit

    • reOpenStores

      protected void reOpenStores(int endReason, long newFileTime)
      Called from the process to close and reopen each datastream in a new file. Probably gets called about once an hour on the hour.

    • getStreamingDataBlocks

      public static ArrayList<PamDataBlock> getStreamingDataBlocks(boolean binaryStore)
      Get a list of data blocks with binary storage. If input parameter is true, then stores which have their binary storage disabled will NOT be included in the list.
      Parameters:
      binaryStore - true if binary storage - blocks with binaryStore flag set false will not be included.
      Returns:
      list of data blocks which have binary storage.

    • getSettingsReference

      public Serializable getSettingsReference()
      Specified by:
      getSettingsReference in interface PamSettings
      Returns:
      The serialisable object that will be stored

    • getSettingsVersion

      public long getSettingsVersion()
      Specified by:
      getSettingsVersion in interface PamSettings
      Returns:
      An integer version number for the settings

    • getFolderName

      public String getFolderName(long timeStamp, boolean addSeparator)
      Get the name of the current storage folder and check the folder exists, if necessary, creating it.
      Parameters:
      timeStamp - current time in milliseconds.
      addSeparator - if true, add a path separator character to the end of the path.
      Returns:
      true if folder exists (or has been created)

    • restoreSettings

      public boolean restoreSettings(PamControlledUnitSettings pamControlledUnitSettings)
      Specified by:
      restoreSettings in interface PamSettings
      Parameters:
      pamControlledUnitSettings -
      Returns:
      true if successful The object performs final checks (if needed) and then casts the settings data pamcontrolledunitSettings.settings into the correct type and uses as required

    • checkCommandLine

      public boolean checkCommandLine()

    • createFileMenu

      public JMenuItem createFileMenu(JFrame parentFrame)
      Overrides:
      createFileMenu in class PamControlledUnit
      Parameters:
      parentFrame - parent frame for the menu
      Returns:
      the file menu item

    • openBinaryFolder

      protected void openBinaryFolder()

    • saveStartSettings

      public boolean saveStartSettings(long timeNow)
      Description copied from interface: PamSettingsSource
      Save the settings in some way or another.
      Specified by:
      saveStartSettings in interface PamSettingsSource
      Parameters:
      timeNow - current time.
      Returns:
      true if settings saved successfully

    • saveEndSettings

      public boolean saveEndSettings(long timeNow)
      Description copied from interface: PamSettingsSource
      Save settings when processing ends. This may just be an update of the settings saves with saveStartSettings, e.g. an end time.
      Specified by:
      saveEndSettings in interface PamSettingsSource
      Parameters:
      timeNow -
      Returns:
      true if saved correctly.

    • getNumSettings

      public int getNumSettings()
      Description copied from interface: PamSettingsSource
      Get the number of different settings within the settings source.
      Specified by:
      getNumSettings in interface PamSettingsSource
      Returns:
      the number of PamSettingsGroups.

    • getSettings

      public PamSettingsGroup getSettings(int settingsIndex)
      Description copied from interface: PamSettingsSource
      Get a specific PamSettingsGroup
      Specified by:
      getSettings in interface PamSettingsSource
      Parameters:
      settingsIndex - index of group
      Returns:
      a settings group

    • getSettingsSourceName

      public String getSettingsSourceName()
      Description copied from interface: PamSettingsSource
      Get a name for the settings source
      Specified by:
      getSettingsSourceName in interface PamSettingsSource
      Returns:
      a name

    • getBinaryStoreSettings

      public BinaryStoreSettings getBinaryStoreSettings()
      Returns:
      the binaryStoreSettings

    • setBinaryStoreSettings

      public void setBinaryStoreSettings(BinaryStoreSettings binaryStoreSettings)
      Parameters:
      binaryStoreSettings - the binaryStoreSettings to set

    • createOfflineDataMap

      public void createOfflineDataMap(Window parentFrame)
      Description copied from interface: OfflineDataStore
      Create a basic map of the data including first and last times and some kind of data/unit time count plus ideally some kind of start and stop time list of where there are gaps in the data.
      Specified by:
      createOfflineDataMap in interface OfflineDataStore

    • listAllFiles

      public List<File> listAllFiles()
      List all the files in the binary storage folder
      Returns:
      List of files.

    • listAllFilesWithPrefix

      public List<File> listAllFilesWithPrefix(String prefix)
      List all the pgdf files in the binary storage folder with the passed prefix
      Returns:
      List of files.

    • listIndexFiles

      public List<File> listIndexFiles()
      List all the index files in the binary storage folder
      Returns:
      List of index files

    • getFileHeaderAndFooter

      public BinaryHeaderAndFooter getFileHeaderAndFooter(File file)
      Run checks on each file.
      Parameters:
      file - file to check.

    • removeMapPoint

      public boolean removeMapPoint(File aFile, ArrayList<PamDataBlock> streams)

    • createMapPoint

      public boolean createMapPoint(File aFile, BinaryHeaderAndFooter bhf, ArrayList<PamDataBlock> streams)
      Create a data map point and add it to the map.

      First the correct data stream (PamDataBlock) must be located then the correct map within that stream. All this searching is necessary since the files will not be read in order so we've no idea up until now which stream it's going to be.

      Parameters:
      bhf - binary header and footer.
      streams - list of data streams
      Returns:
      true if stream and map found and map point added.

    • readHeaderAndFooter

      public BinaryHeaderAndFooter readHeaderAndFooter(File file)
      Public version of readHeadAndFoot(File) method
      Parameters:
      file - to read
      Returns:
      header and footer packed up together.

    • findIndexFile

      public File findIndexFile(File dataFile, boolean checkExists)
      Find the index file to match a given data file.
      Parameters:
      dataFile - data file.
      checkExists - check teh file exists and if it doens't return null
      Returns:
      index file to go with the data file.

    • findNoiseFile

      public File findNoiseFile(File dataFile, boolean checkExists)
      Find the noise file to match a given data file.
      Parameters:
      dataFile - data file.
      checkExists - check the file exists and if it doens't return null
      Returns:
      index file to go with the data file.

    • makeIndexFile

      public File makeIndexFile(File dataFile)
      Create an index file (pgdx) name from a data file (pgdf) file name
      Parameters:
      dataFile - data file name
      Returns:
      index file name

    • swapFileType

      public File swapFileType(File dataFile, String newType)
      Swap a file type from whatever is on the end of datafile to newType
      Parameters:
      dataFile - existing data file, probably .pgdf
      newType - new file type (should not have the . on)
      Returns:
      new file with changed type.

    • rewriteIndexFile

      public boolean rewriteIndexFile(PamDataBlock dataBlock, OfflineDataMapPoint offlineDataMapPoint)
      rewrite the index file.
      Specified by:
      rewriteIndexFile in interface OfflineDataStore
      Parameters:
      dataBlock -
      offlineDataMapPoint -
      dmp -
      Returns:

    • rewriteIndexFile

      public boolean rewriteIndexFile(PamDataBlock dataBlock, OfflineDataMapPoint offlineDataMapPoint, File indexFile)
      Parameters:
      dataBlock -
      offlineDataMapPoint -
      indexFile -
      Returns:

    • listDataFiles

      public void listDataFiles(ArrayList<File> fileList, File folder, PamFileFilter filter)
      List all data files - get's called recursively
      Parameters:
      fileList - current fiel list - get's added to
      folder - folder to search
      filter - file filter

    • getDataSourceName

      public String getDataSourceName()
      Description copied from interface: OfflineDataStore
      Get the data source name
      Specified by:
      getDataSourceName in interface OfflineDataStore
      Returns:
      data source name

    • loadData

      public boolean loadData(PamDataBlock dataBlock, OfflineDataLoadInfo offlineDataLoadInfo, ViewLoadObserver loadObserver)
      Description copied from interface: OfflineDataStore
      Load data for a given datablock between two time limits.
      Specified by:
      loadData in interface OfflineDataStore
      Parameters:
      dataBlock - datablock owner of the data
      loadObserver -
      Returns:
      true if load successful.

    • loadData

      public boolean loadData(PamDataBlock dataBlock, BinaryOfflineDataMapPoint mapPoint, long dataStart, long dataEnd, BinaryDataSink dataSink)
      Load the data from a single file.

      Generally, PAMGUARD will use the above function which loads between two times, but for some offline analysis tasks, it's convenient to scroll through file at a time in which case this function can be used.

      Parameters:
      dataBlock - PamDataBlock to receive the data
      dataStart - data start
      dataEnd - data end
      binaryOfflineDataMapPoint - data map point
      Returns:
      true if data were loaded. false if out of memory in which case the calling load function will drop out of the loop over files.

    • unpackAnnotationData

      protected void unpackAnnotationData(int fileVersion, PamDataUnit createdUnit, BinaryObjectData binaryObjectData, BinaryDataSink dataSink)
      Unpack annotation data.
      Parameters:
      createdUnit -
      binaryObjectData -
      dataSink -

    • saveData

      public boolean saveData(PamDataBlock pamDataBlock)
      Saves data in a PamDataBlock.

      First scans all data in the pamDataBlock and works out which files actually need updating based on info in their DataUnitFileInformation then rewrites those files.

      Note that data from many files may be in memory and it's also possible that files are only partially read in, in which case, it will be necessary to partially take data from the old file and partially from the stuff in memory !

      Specified by:
      saveData in interface OfflineDataStore
      Parameters:
      pamDataBlock - data block holding the data.
      Returns:
      true if saved Ok.

    • pamClose

      public void pamClose()
      Description copied from class: PamControlledUnit
      Called when PAMGUARD is finally closing down so that a module may free any remaining resources (e.g. files or COMM ports).
      Overrides:
      pamClose in class PamControlledUnit

    • isDataBlockBinaryOut

      public static boolean isDataBlockBinaryOut(PamDataBlock pamDataBlock)
      Convenience function to determine whether a data block is beiung written to binary files
      Returns:
      only true if binary store is present and the data block subscribes to it.

    • findBinaryStoreControl

      public static BinaryStore findBinaryStoreControl()
      Find the binary store
      Returns:
      binary storage controller, or null if no binary storage module loaded.

    • getBinaryUnitType

      public static String getBinaryUnitType()
      Get the unit type for the binary store.
      Returns:
      the binary store unit type.

    • getDatagramManager

      public DatagramManager getDatagramManager()
      Specified by:
      getDatagramManager in interface OfflineDataStore
      Returns:
      the datagramManager

    • rebuildFileFooter

      public BinaryFooter rebuildFileFooter(File file)
      Reads the data objects in the passed file, and records the first/last UID, times and sample numbers. Uses the information to create a new BinaryFooter object, and then passes that back to the calling function. Note that this method sets the BinaryFooter's FileEndReason as END_CRASHED, which is usually the case if the footer needs to be rebuilt.
      Parameters:
      file - the binary file to examine
      Returns:
      a BinaryFooter object, or null if there was an error

    • getGUI

      public PamControlledUnitGUI getGUI(int flag)
      Get the GUI for the PAMControlled unit. This has multiple GUI options which are instantiated depending on the view type.
      Overrides:
      getGUI in class PamControlledUnit
      Parameters:
      flag - . The GUI type flag defined in PAMGuiManager.
      Returns:
      the GUI for the PamControlledUnit unit.

    • getModuleStatus

      public ModuleStatus getModuleStatus()
      Description copied from class: PamControlledUnit
      Quick convenient way of getting the module status.
      Overrides:
      getModuleStatus in class PamControlledUnit
      Returns:
      Module Status or null if no status manager present.

    • getBackupInformation

      public BackupInformation getBackupInformation()
      Overrides:
      getBackupInformation in class PamControlledUnit

    • getNoiseStore

      public NoiseStoreType getNoiseStore()
      Store noise in pgdf or pgnf files.
      Returns:
      where noise data are to be stored.

    • getStoreStatus

      public StoreStatus getStoreStatus(boolean getDetail)
      Description copied from interface: DataOutputStore
      Get the store status, i.e. does it exist, does it contain data, if so over what date range, etc.
      Specified by:
      getStoreStatus in interface DataOutputStore
      Parameters:
      getDetail -
      Returns:

    • deleteDataFrom

      public boolean deleteDataFrom(long timeMillis)
      Description copied from interface: DataOutputStore
      Delete all data from a given time, in all data streams.
      Specified by:
      deleteDataFrom in interface DataOutputStore
      Parameters:
      timeMillis - time to delete from (anything >= this time)
      Returns:
      true if it seems to have worked OK. False if any errors (e.g. database or file system error).

    • getOfflineState

      public int getOfflineState()
      Description copied from class: PamControlledUnit
      Get the offline state of this module. This can generally be idle, but can be a higher state when map making at startup and when running an offline task.
      Overrides:
      getOfflineState in class PamControlledUnit
      Returns:

    • getDataLocation

      public String getDataLocation()
      Description copied from interface: OfflineDataStore
      Get the data location. This may be a specific file, or might be a folder if data are in many files, a URI, etc.
      Specified by:
      getDataLocation in interface OfflineDataStore
      Returns:
      store locations

    • getInegrityChecker

      public DataIntegrityChecker getInegrityChecker()
      Description copied from interface: DataOutputStore
      Get a data integrity checker. This can be called at startup to see if there is a problem.
      Specified by:
      getInegrityChecker in interface DataOutputStore
      Returns:

    • getBinaryStoreProcess

      public BinaryStoreProcess getBinaryStoreProcess()
      Returns:
      the binaryStoreProcess