Release notes¶

Development¶

Upgrading from 3.0¶

Make sure you are upgrading from 3.0
Fix any time zone errors, as described in the appropriate section below.
Backup the database.
Login as a superuser and go to the dashboard.
Go to “Sites” and make sure they’re set correctly.
In the settings, make sure SITE_ID is set correctly.
In the settings, remove setting ENHYDRIS_OPEN_CONTENT and instead set ENHYDRIS_DEFAULT_PUBLICLY_AVAILABLE and ENHYDRIS_ENABLE_TIMESERIES_DATA_VIEWERS. Note that, during migration, the new publicly_available attribute will be set on existing time series to the value of ENHYDRIS_DEFAULT_PUBLICLY_AVAILABLE.
Update the database with python manage.py migrate. This will put all stations to the site specified with SITE_ID, and will add all users to the group whose name is the domain of the current site (the group will be created automatically if it does not exist).
If you have been using a single database to power many sites, then:
- In the settings, make sure ENHYDRIS_SITES_FOR_NEW_STATIONS is set correctly. Restart the server if necessary.
- Logon as a superuser and go to the dashboard
- Go to each of the stations that used to be specified by ENHYDRIS_SITE_STATION_FILTER and make sure the “Sites” field is set correctly.
- If any users need to be able to log on to a different site from the one where you performed the database update, go to each of these users and put them in the appropriate groups.
Restart the server.

Changes from 3.0¶

Many sites¶

Functionality for serving many sites from a single database has been added. Accordingly, the setting ENHYDRIS_SITE_STATION_FILTER has been abolished and replaced with ENHYDRIS_SITES_FOR_NEW_STATIONS. For more information, see Managing domains.

Time zones¶

Enhydris 3.0 has a serious bug. TimeseriesGroup has a time_zone attribute. When a time series is being uploaded through the Django admin or through the Enhydris API, the timestamps are naive and are considered to be in the timeseries group’s time zone, and are converted to UTC for storage. Conversely, when a time series is being retrieved (e.g. downloaded), the timestamps are converted to the timeseries group’s time zone.

While this behaviour looks reasonable, it is not what was intended. enhydris.models.TimeseriesGroup.time_zone was intended to be informational. The intention was that naive timestamps would be stored naively, and that if you stored 2022-10-10 19:30, then if you retrieved it it would be 2022-10-10 19:30. But this is not how it works. Timestamps are stored not naively but in UTC. If the group’s time zone is EET and you store that timestamp and then you change the group’s time zone to CET and you retrieve the timestamp again, you’ll get 18:30 instead. If you have an automatically updated time series with time zone MSK and one day you see it and say “hey, that’s wrong!” and you change it to CET, it will continue to update the data but now the dates will mean something different.

In order to fix these problems, the enhydris.models.TimeseriesGroup.time_zone attribute has been abolished. Attribute enhydris.models.Gentity.display_timezone has been added, which only affects how timestamps are displayed (or downloaded). When storing a time series (though the admin or the API), the time zone of the timestamps has to be explicitly specified.

In order to upgrade from 3.0, first you need to inspect all your time series groups with respect to the time zone specified. If for some time series groups the time zone is wrong (but the timestamps appear to be correct when downloading the time series), then you need to fix it in this way for each time series group that has the problem:

Download all time series of the group.
Fix the time zone of the group.
Upload all the downloaded time series back to Enhydris, discarding existing data.

If you discover errors after upgrading, fixing them is similar:

Download the time series that have the problem.
Upload the time series, specifying an appropriate time zone.

Permissions¶

Several things have been changed in the permissions model. Setting ENHYDRIS_OPEN_CONTENT has been abolished and replaced with the new settings ENHYDRIS_DEFAULT_PUBLICLY_AVAILABLE and ENHYDRIS_ENABLE_TIMESERIES_DATA_VIEWERS. These two settings together can do much more than what ENHYDRIS_OPEN_CONTENT did. Another important difference is that these new settings apply only to time series data, not to gentity files, so gentity files are now always publicly available.

In addition, setting ENHYDRIS_AUTHENTICATION_REQUIRED has been added and can make Enhydris fully closed.

Version 3.0¶

Released on 17 August 2021.

Upgrading¶

You may only upgrade from version 2.1 (version 2.1 only exists to facilitate transition to 3.0, and it is otherwise not used; the old stable Enhydris version is 2.0). The procedure is this:

Make sure you are running version 2.0 (any release will do).

Backup the database.

Make sure you have read and understood the list of changes from 2.0 below, as some of these changes may require manual intervention or automatically do things you might not want.
Update the repository:
git fetch origin
Shut down the running service.
Install version 2.1 and migrate:
git checkout 2.1
python manage.py migrate
Empty the migrations table of the database for the hcore app:
python manage.py migrate --fake hcore zero
(This step is optional because in 3.0 the hcore app goes away and is replaced by enhydris. You can omit it in case you need to go back or execute it if you want a cleaner database.)
Install TimescaleDB and restart PostgreSQL. You don’t need to create the extension in the database; the Django migrations will do so automatically. See “TimescaleDB” in the “Changes from 2.0” below for more information.

In the settings, make sure SITE_ID, LANGUAGE_CODE and PARLER_LANGUAGES are set properly. See “Multilingual contents” in the “Changes from 2.0” below for more information.
Install version 3.0:
git checkout 3.0
pip install -r requirements.txt
If your settings file has been in enhydris/settings/, you need to create a settings file in enhydris_project/settings/, as this location has changed.
Empty the migrations table for the registration app:
python manage.py migrate --fake registration zero
If you fail to perform this step, you may get the message ‘relation “registration_registrationprofile” does not exist’ or similar. The exact cause is not known, however lots of things have changed regarding the registration system.
Execute migrations:
python manage.py migrate --fake-initial
If some migrations succeed and there is a failure later, you should probably omit the –fake-initial parameter in subsequent attempts. There is, notably, a possibility of an error related to registration happening (as described in the previous step); in such a case, repeat the previous step and then re-execute the above migration command (possibly without –fake-initial).
Remove obsolete settings from the settings file.

Start the service.

Create and start a celery service.

Changes from 2.0¶

Time series groups¶

In 2.0, a station has time series. Now it has time series groups and each group consists of time series with essentially the same kind of data but in a different time step or in a different checking status. For example, if you have a temperature sensor that measures temperature every 10 minutes, then you will have a “temperature” time series group, which will contain the initial time series, and it may also contain the checked time series, the regularized time series, the hourly time series, etc. (If you have two temperature sensors, you’ll have two time series groups.)

We avoid showing the term “time series group” to the user (instead, we are being vague, like “Data”, or we might sometimes use “time series” when we actually mean a time series group). Sometimes we can’t avoid it though (notably in the admin).

Each time series in the group has a “type” (which is enumerated): it can be initial, checked, regularized, or aggregated.

During database upgrade, unless enhydris-autoprocess is installed, each existing time series goes in a separate group, and it is assumed it is the initial. In many cases, this is the correct assumption. If enhydris-autoprocess is installed, the database upgrade attempts to find out which time series is the initial, which is checked, and which is aggregated (however enhydris-autoprocess did not exist for Enhydris 2.0, so this applies only to installations of Enhydris development versions).

TimescaleDB¶

We now store time series data in the database using TimescaleDB. Before that, time series data was stored in files in the filesystem, in CSV format, one file per time series.

The location where the files were being stored was specified by setting ENHYDRIS_TIMESERIES_DATA_DIR. This setting has now been abolished.

The size of your database will increase considerably. The increase in size maybe eight times the size of ENHYDRIS_TIMESERIES_DATA_DIR. Make sure you have the available disk space. Also make sure that your PostgreSQL backup strategy can handle the increased size of the database.

When executing the migrations, the time series data will be read from the files and entered to the database. The files will not be removed.

The migration will only work if the PostgreSQL server runs in the same machine as Enhydris. This is because, in order to speed up the importing of the data to the database, the files are read directly by the database server using the SQL COPY ... FROM command. See the code for the migration for more details.

Since a single transaction could be too much for the entire importing (it would use lots of space and be very slow), the transaction is committed for each time series. This means that if you interrupt the migration, the database will contain some, but not all, records. Attempting to run the migration a second time will therefore fail. In such a case, before attempting to re-run the migration, empty the table like this:

echo "DELETE FROM enhydris_timeseriesrecord" | ./manage.py dbshell

In addition, to speed up importing of the data, table constraints and indexes are created after the data is imported. This may mean that it could fail after importing if there are duplicate dates in the timeseries data. This can happen because of an old bug. In such a case, reverse the migration (empty the table as above if needed), run the following inside the ENHYDRIS_TIMESERIES_DATA_DIR directory to find the problems, fix them and re-run the migration:

for x in *; do
    a=`uniq -w 16 -D $x`
    if [ -n "$a" ]; then
        echo ========= $x
        echo "$a"
        echo
    fi
done

As an order of magnitude, conversion of the data should take something like 40 minutes per GB of ENHYDRIS_TIMESERIES_DATA_DIR storage space, but of course this depends on several factors. Roughly half of this time will be for the importing of the data, and another half for the creation of the indexes (however these times might not actually be linear).

Celery¶

In 2.0, nothing was done asynchronously. In 3.0, the uploading of time series data through the site (not through the Web API) is performed asynchronously, i.e. the user receives a message that the time series data are about to be imported, and he is emailed when importing finishes.

Therefore, a Celery service must be running on the server.

Some add-on applications, like enhydris-synoptic and enhydris-autoprocess, also use Celery.

Multilingual contents¶

The way we do multilingual database contents has changed.

We were using a hacky system where two languages were offered; e.g. there was Gentity.name and Gentity.name_alt, where the latter was the name in the “alternative” language. This system, rather than a “correct” one that uses, e.g., django-parler, was more trouble than it was worth, therefore all fields ending in _alt have been abolished.

In the new Enhydris version, several lookups, such as variable names, are multilingual using django-parler. However, station and timeseries names and remarks, event reports, etc. (i.e. everything a non-admin user is expected to enter), are not multilingual. The idea is that a station in Greece will have a Greek name, and this does not need to be transliterated. The rationale is the same as for OSM’s-avoid-transliteration rule: transliterations can be automated, and having users enter them manually would only create noise in the database. There may be valid cases for translation (e.g. when the name of a station is “bridge X”, or translation of remarks), but users generally don’t enter translations so we haven’t developed this functionality yet.

For the case of fields that are untranslated in the new version, while upgrading, for each row, whichever of fieldname and fieldname_alt is nonempty will be used for fieldname. If both are nonempty and they are single-line fields, “value of fieldname [value of fieldname_alt]” will be used for fieldname, i.e. the value of fieldname_alt will be appended in square brackets. If the number of characters available is insufficient an error message will be given and the upgrade will fail. If both fields are nonempty and they are multi-line fields such as TextField, they will be joined together separated by \n\n---ALT---\n\n.

For the case of lookups translated with django-parler, fieldname becomes the main language (set by LANGUAGE_CODE or PARLER_DEFAULT_LANGUAGE_CODE), and fieldname_alt becomes the second language, i.e. the second entry of PARLER_LANGUAGES. If PARLER_LANGUAGES has fewer than two languages, then the conversion described in the previous paragraph takes place.

(In fact, because abolishing of _alt fields was decided and implemented several months before deciding to use django-parler on lookups, the migration system will convert everything to unilingual as described above, and then it will convert lookups back to multilingual.)

Before upgrading the database, it is important to set SITE_ID, LANGUAGE_CODE, and PARLER_LANGUAGES. SITE_ID is probably already set, probably by the default Enhydris settings. Keep it as it is. Set LANGUAGE_CODE to the language that corresponds to the main language of the site, i.e. the one to which lookup descriptions not ending in _alt correspond. Finally, set PARLER_LANGUAGES as follows:

PARLER_LANGUAGES = {
    SITE_ID: [
      {"code": LANGUAGE_CODE},
      {"code": "specify_your_second_language_here"},
    ],
}

Because of what is likely a bug in django-parler (at least 2.0), it is important to use SITE_ID as the key and not None.

Geographical areas¶

Each station (and more generally each Gentity) used to have three foreign keys to water basins, water divisions, and political divisions (the latter were hierarchical, being countries at the top level). This is no longer the case. Water basins, water divisions, and political divisions have been abolished. Instead, there is a mere Garea entity, that can belong in a category. You create as many categories as you want (countries, water basins, prefectures, whatever you like) and you upload a shapefile of them (it’s mandatory that they have a geometry).

There is no foreign key between stations (or other Gentities) and Gareas. To find which stations are in a Garea, the system does a point-in-polygon query.

The upgrade will delete all existing water basins, water divisions, and political divisions, and all existing relationships between them. This change is non-reversible. It will not create any Gareas. You can use the admin to upload Gareas.

Other changes¶

The Web API has been reworked. Applications using the Enhydris 2.0 web API won’t work unchanged with 3.0.
The templates have been refactored. Applications and installations with custom templates or templates inheriting the Enhydris templates may need to be modified.
Instruments have been abolished. Upgrading requires the database to not have any instruments. If you try to upgrade and there are instruments, it will give you an error message with instructions on how to empty the instruments table.
GentityGenericData and GentityAltCode have been abolished, as they were practically not being used in any of the known installations. Upgrading requires the tables to be empty; if not, upgrading will stop with an error message. Make sure the tables are empty before upgrading.
Gpoint.point has been renamed to Gpoint.geom.
Stations now must have co-ordinates, i.e. the related database field gpoint.geometry (formerly gpoint.point) is not null. If you have any stations with null co-ordinates, they will be silently converted to latitude zero and longitude zero during upgrading.
The time step is now stored as a pandas “frequency” string, e.g. “10min”, “H”, “M”, “Y”. The TimeStep model does not exist any more. The timestamp_rounding, timestamp_offset and interval_type properties have been abolished. During the database upgrade, they are simply dropped.
SQLite is no longer supported.
The fields approximate (used to denote that a station’s location has been assigned roughly) and asrid (altitude SRID) have been abolished. The field srid has been renamed to original_srid.
The field Gentity.short_name has been renamed to Gentity.code.
Station types have been abolished. Stations now don’t have a type. The related information previously stored in the database will be deleted in the upgrade.
Stations can now only have a single overseer, specified as a text field. Upgrading will convert as needed, and it will also delete any unreferenced Person objects.
The field Station.is_automatic has been abolished.
The database fields copyright_years and copyright_holder have been abolished. The database upgrade will remove them and any information stored in them will be lost. Accordingly, the setting ENHYDRIS_DISPLAY_COPYRIGHT_INFO has been abolished.
OpenLayers has been replaced with Leaflet. Accordingly, the form of the ENHYDRIS_MAP_BASE_LAYERS setting has been changed and the setting ENHYDRIS_MAP_DEFAULT_BASE_LAYER has been added.
The setting ENHYDRIS_SITE_CONTENT_IS_FREE has been abolished. ENHYDRIS_TSDATA_AVAILABLE_FOR_ANONYMOUS_USERS has been renamed to ENHYDRIS_OPEN_CONTENT. Several other settings that were rarely being used have been abolished or renamed.

Version 2.0¶

Upgrading¶

You can upgrade directly from versions 0.8 and later. If you have an older version, first upgrade to 0.8.

Enhydris is no longer pip-installable. Instead, it is a typical Django application with its manage.py and all. Install it as described in Installation and configuration and execute the database upgrade procedure:

python manage.py migrate

Changes from 1.1.2¶

Now a normal Django project, no longer pip-installable.
Django 1.11 and only that is now supported.
A favicon has been added.
Several bugs have been fixed. Notably, object deletions are confirmed.

Changes in 2.0 microversions¶

Version 2.0.1 removes EMAIL_BACKEND from the base settings and leaves the Django default (this broke some production sites that did not specify EMAIL_BACKEND and were expecting the Django default.)
Version 2.0.2 adds pagination to the list of stations and requires a Django-1.11-compatible version of django-simple-captcha.
Version 2.0.3 fixes an undocumented CSV view that sends you a zip file with stations, instruments and time series in CSV when you add ?format=csv to a stations list URL. Apparently this had been broken since version 1.0.
Version 2.0.4 fixes several crashes.

Version 1.1¶

Upgrading¶

There are no database migrations since version 0.8, so you just need to install the new version and you’re good to go.

Changes in 1.1 microversions¶

Version 1.1.0 changes an internal API; enhydris.hcore.models.Timeseries.get_all_data() is renamed to enhydris.hcore.models.Timeseries.get_data() and accepts arguments to specify a start and end date.
Version 1.1.1 puts the navbar inside a {% block %}, so that it can be overriden in custom skins.
Version 1.1.2 fixes two bugs when editing time series: appending wasn’t working properly, and start and end dates were shown as editable fields.

Version 1.0¶

Overview¶

This version has important internal changes, but no change in functionality (except for the fix of a minor bug, that the time series chart would apparently “hang” with a waiting cursor showing for ever when a time series was empty). These important changes are:

Python 3 is now supported, and there is no more support for Python 2.
Pthelma is not used anymore; instead, there is a dependency on pandas and on the new pd2hts module.

Upgrading from 0.8¶

Make sure you are running Enhydris 0.8. Discard your virtualenv and follow the Enhydris installation instructions to install the necessary operating system packages and install Enhydris in a new Python 3 virtualenv. You don’t need to change anything in the configuration or perform any database migration.

Changes in 1.0 microversions¶

When downloading time series and specifying a start date, the resulting time series could start on a slightly different start date because of some confusion with the time zone. The bug was fixed in 1.0.1.
Gentity files could not be downloading because of a bug in the downloading code. Fixed in 1.0.2.

Version 0.8¶

Overview¶

The time series data are now stored in files instead of in database blobs. They are stored uncompressed, which means that much more disk space is consumed, but it has way more benefits. If disk space is important to you, use a file system with transparent compression.
Experimental spatialite support.

Upgrading from 0.6¶

The upgrade procedure is slightly complicated, and uses the intermediate Enhydris version 0.7, which exists only for this purpose.

(Note for developers: the reason for this procedure is that the migrations have been reset. Previously the migrations contained PostgreSQL-specific stuff.)

The upgrade procedure is as follows:

Backup your database, your media files, and your configuration (you are not going to use this backup unless something goes wrong and you need to restore everything to the state it was before).
Make sure you are running Enhydris 0.6.
Follow the Enhydris 0.8 installation instructions to install Enhydris in a new virtualenv; however, rather than installing Enhydris 0.8, install, instead, Enhydris 0.7, like this:
```
pip install 'enhydris>=0.7,<0.8'
```
Open your settings.py and add the configuration setting ENHYDRIS_TIMESERIES_DATA_DIR. Make sure your server has enough space for that directory (four times as much as your current database, and possibly more), and that it will be backing it up.
Apply the database upgrades:
```
python manage.py migrate
```

Install Enhydris 0.8:

pip install --upgrade --no-deps 'enhydris>=0.8,<0.9'

Have your database password ready and run the following to empty the django_migrations database table:
```
python manage.py dbshell
delete from django_migrations;
\q
```
Repopulate the django_migrations table:
```
python manage.py migrate --fake
```

Version 0.6¶

Overview¶

The skin overhaul has been completed.
The confusing fields “Nominal offset” and “Actual offset” have been renamed to “Timestamp rounding” and “Timestamp offset”. For this, pthelma>=0.12 is also required.
Data entry of station location has been greatly simplified. The user now merely specifies latitude and longitude, and only if he chooses the advanced option does he need, instead, to specify ordinate, abscissa, and srid.
Several bugs have been fixed.

Backwards incompatible changes¶

The is_active fields have been removed.

Stations and instruments had an is_active field. Apparently the original designers of Enhydris thought that it would be useful to make queries of, e.g., active stations, as opposed to all stations (including obsolete ones).

However, the correctness of this field depends on the procedures each organization has. Many organizations don’t have a specific procedure for obsoleting a station; a station merely falls out of use (e.g. an overseer stops working and (s)he is never replaced). Therefore, it is unlikely that someone will go and enter the correct value in the is_active field. Even if an organization does have processes that could ensure correctness of the field, they could merely specify an end date to a station or instrument, and therefore is_active is superfluous.

Indeed, in all Hydroscope databases, the field seems to be randomly chosen, and in openmeteo.org it makes even less sense, since it is an open database whose users are expected to merely abandon their stations and not care about “closing” them properly.

Therefore, the fields have been removed. However, the database upgrade script will verify that they are not being used before going on to remove them.

Upgrading from 0.5¶

Backup your database (you are not going to use this backup unless something goes wrong and you need to restore everything to the state it was before).
Make sure you are running the latest version of Enhydris 0.5 and that you have applied all its database upgrades (running python manage.py migrate should apply all such upgrades, and should do nothing if they are already applied).
Install 0.6 and execute the database upgrade procedure:
```
python manage.py migrate
```

Changes in 0.6 microversions¶

Added some explanatory text for timestamp rounding and timestamp offset in the time series form (in 0.6.1).

Version 0.5¶

Overview¶

There has been a huge overhaul of the Javascript.
The map base layers are now configurable in settings.py.
The map has been simplified and now uses OpenLayers 2.12.
The “advanced search” has been removed. Instead, it is possible to perform advanced searches by writing the appropriate code in the single search box. The “Search tips” link beside the search box provides instructions.
The skin has been modernized and simplified and uses Bootstrap. This is work in progress.
The installation procedure has been greatly simplified.
Django 1.8 support.

Backwards incompatible changes¶

Only supports Python 2.7 and Django 1.8.
Removed apps hchartpages and dbsync. These are expected to be replaced by independent applications in the future, but no promises are made. Enhydris is to become a small, reliable and well-maintained core.

Upgrading from 0.2¶

Version 0.5 contains some tricky database changes. The upgrade procedure is slightly complicated, and uses the intermediate Enhydris version 0.3, which exists only for this purpose.

(Note for developers: the reason for this procedure is that hcore used to have a foreign key to a dbsync model. As a result, the initial Django migration listed dbsync as a dependency, making it impossible to remove dbsync.)

The upgrade procedure is as follows:

Backup your database (you are not going to use this backup unless something goes wrong and you need to restore everything to the state it was before).
Make sure you are running the latest version of Enhydris 0.2 and that you have applied all its database upgrades (running python manage.py migrate should apply all such upgrades, and should do nothing if they are already applied).
Follow the Enhydris 0.5 installation instructions to install Enhydris in a new virtualenv; however, rather than installing Enhydris 0.5, install, instead, Enhydris 0.3, like this:
```
pip install 'enhydris>=0.3,<0.4'
```

Apply the database upgrades:

python manage.py migrate --fake-initial

Install Enhydris 0.5. The simplest way (but not the safest) is this:
```
pip install --upgrade --no-deps 'enhydris>=0.5,<0.6'
```
However, it is best to discard your Enhydris 0.3 virtualenv and create a new one, in which case you would install Enhydris 0.5 like this:
```
pip install 'enhydris>=0.5,<0.6'
```
Have your database password ready and run the following to empty the django_migrations database table:
```
python manage.py dbshell
delete from django_migrations;
\q
```
Repopulate the django_migrations table:
```
python manage.py migrate --fake
```

Changes in 0.5 microversions¶

Removed embedmap view (in 0.5.1)
Removed example_project, which was used for development instances; instead, added instructions in README.rst on how to create one (in 0.5.1).
Fixed internal server error when editing station with ENHYDRIS_USERS_CAN_ADD_CONTENT=True (in 0.5.2).
Since 0.5.3, Enhydris depends on pthelma<0.12, since pthelma 0.12 has a backwards incompatible change.

Version 0.2¶

Changes¶

There have been too many changes to list here in detail. The most important ones (particularly those affecting backwards compatibility) are:

Removed apps hrain, gis_objects, contourplot, hfaq, contact. hfaq and contact should be replaced with flatpages. hrain, gis_objects, and contourplot are not supported any more. If they are included again in the future, they will be maintained separately as distinct applications. Enhydris is to become a small, reliable and well-maintained core.
Removed front page; front page is now station list
Compatible with Django 1.5 and 1.6.

Upgrading from 0.1¶

Essentially you are on your own. It’s likely that just installing Enhydris 0.2 and executing python manage.py migrate will do the trick. Don’t forget to backup your database before attempting anything!