For developers

Loggers are machines to which sensors are connected; they log the measurements of the sensors. In order to avoid confusion with Python Logger objects, here and in the code we call them meteologgers. But the user should not be bothered with that, so user documentation uses “loggers” for meteologgers.

MeteologgerStorage objects

In its simplest and most usual form, a “meteologger storage” is a text file to which the meteologger software appends records. Each line of the file has a timestamp and the values of the various measurements.

Some kinds of meteologgers (or meteologger software), however, use different kinds of storage. They may be automatically starting a new file daily or monthly. In such cases, the “meteologger storage” is a directory that has these files.

The MeteoLoggerStorage class is an abstract base class that is meant to be subclassed. It provides functionality that is common to all kinds of meteologger storages. Since each make of meteologger, or even each type of software used to unload data from a meteologger, provides a different kind of meteologger storage, MeteologgerStorage subclasses are specialized, each to a specific kind of meteologger storage.

class MeteologgerStorage(parameters[, logger=None])

A MeteologgerStorage instance should never be constructed; instead, a subclass should be constructed; however, the call for constructing any subclass has the same arguments. parameters is a dictionary containing values for configuration parameters such as path and fields. The logger argument is a Logger object to which error, progress, and debugging information is written.

When MeteologgerStorage is initialized, it checks the correctness of parameters and raises ConfigurationError if anything’s wrong. See get_required_parameters() and get_optional_parameters() for more information.

The following MeteologgerStorage attributes are initialized from parameters:

path

The pathname to the storage; a filename or directory name.

null

A representation of values that will be treated as NaN (or null) (see Usage for more information).

(nullstr is a deprecated synonym of null.)

fields

A comma-separated list of integers representing the ids of the time series groups to which the fields correspond; a zero indicates that the field is to be ignored. The first number corresponds to the first field after the date (and other fixed fields, such as the possible subset identifier; which are those fields depends on the file format, that is, the specific MeteologgerStorage subclass) and should be the id of the corresponding time series group, or zero if the field is dummy; the second number corresponds to the second field after the fixed fields, and so on.

nfields_to_ignore

This is used only in the simple format; it’s an integer that represents a number of fields before the date and time that should be ignored. The default is zero. If, for example, the date and time are preceded by a record id, set nfields_to_ignore=1 to ignore the record id.

subset_identifiers

This is used only on some MeteologgerStorage subclasses. Some file formats mix two or more sets of measurements in the same file; for example, there may be ten-minute and hourly measurements in the same file, and for every 6 lines with ten-minute measurements there may be an additional line with hourly measurements (not necessarily the same variables). Such files have one or more additional distinguishing fields in each line, which helps to distinguish which set it is. We call these fields, which depend on the specific file format, the subset identifiers.

MeteologgerStorage (in fact its subclass) processes only one set of lines each time, and subset_identifiers specifies which subset it is. subset_identifiers is a comma-separated list of identifiers, and will cause MeteologgerStorage (in fact its subclass) to ignore lines with different subset identifiers.

delimiter
decimal_separator
date_format

Some file formats may be dependent on regional settings; these formats (i.e. these MeteologgerStorage subclasses) support delimiter, decimal_separator, and date_format. date_format is specified in the same way as for strftime.

MeteologgerStorage also has the following methods and properties:

timeseries_group_ids

A list of time series group ids. This is extracted from fields (zeros are ignored).

get_recent_data(ts_group_id, ts_id, after_timestamp)

Read the storage and extract the last part of the time series that is specified by ts_group_id and ts_id; specifically, provide the part that is more recent than after_timestamp (an aware datetime object). Returns that part of the time series as a pandas dataframe.

get_recent_data() will actually extract the last part of all time series from storage, but only return the data for the requested time series. It will cache the rest and have them ready to return for subsequent calls. However, if in subsequent calls after_timestamp is earlier than in the first call, it will need to re-extract the time series from storage. Therefore, for better performance, use the smallest after_timestamp first.

_raise_error(line, msg, filename=None)

This is only meant to be used internally, i.e. called by subclasses whenever an error is found in a data file. The method raises an exception. line and msg are strings used in the error message. filename is an optional parameter specifying the name of the file where the error occurred.

_is_null(value)

This is only meant to be called by subclasses whenever they want to check whether a given value is null.

MeteologgerStorage subclasses need to define the following methods:

_subset_identifiers_match(line)

Return True if line matches the subset_identifiers. The base method always returns True, and subclasses only need to redefine it if the file format has subsets.

_extract_timestamp(line)

Parse line and extract and return the date and time as an aware datetime object.

_extract_value_and_flags(line, seq)

Extract the value and flags in sequence seq from line, and return it as a tuple. seq=1 is the first field after the fixed field, and so on (similar to fields).

get_required_parameters()

Return a set of parameters that are required. The base method returns {"path", "storage_format", "file_fields"} and must be overridden to add items to the list if the subclass requires more parameters.

get_optional_parameters()

Return a list of optional parameters. The base method returns an empty set and must be overridden if the subclass allows a different set.