"Incorrect" rain readings.

[John B]

I run two web-sites. www.blaxlandweather.com and www.johnsblaxlandweather.id.au - they are not linked.
The first one runs on an Australian meteorological day of 0900 to 0900 and my own coding.
The second uses the Meteotemplate calender day and Jáchym's Meteotemplate software. I am advised that it is not possible to change this.

Each of them uses the same weather station information and Weather Display software as its data source.

The Weather Display program is set to use the 0900 day for averages and the re-setting of daily data.

The .com site uses Weather Display's clientraw series of text files. The id.au site uses the Weather Display API.

The .com site has a MySQL data-base but it is no longer updated. The id.au site has an active MySQL data-base.

It seems to me that there is some sort of conflict between the two sites, possibly to do with the use of the different 'days'. Perhaps Brian's API has not taken into account that the user's settings can differ? In which case I suppose I should address this problem on his forum. I flipped a coin and am posting here.

An example: Today 2017-10-20

Station Data shows 11.9mm of rain (since the day started at midnight). It shows a maximum gust of wind at 20.4Km/h at 12:35
The Weather Display program shows 11.4mm of rain since 09:00 and a max wind gust of 21.2Km/h at 12:29.
Ignoring the wind, although the readings are probably incorrect, the rain readings are wrong. Yesterday's rain reading in the .com site are correct at 11.9mm. Since 09:00 we have had Oh, darn it the same amount. Aaargh. Anyway, if you look at the attached image it should be 23.8mm.*

This causes the rain information in MT to be grossly incorrect. I also have a normal rain gauge I use to check the veracity of the readings coming from the station tipping unit. The three readings for the year-to-date period are:

Station: 666.8 mm
MT: 874.4 mm
Manual: 669.2 mm

I have changed temporarily the Weather Display day to the calendar day, but I'll have to await the next rain to see if it changes the records. I'm loathe to leave it like that as it conflicts with our official 'day'.

Am I missing something somewhere?

The image will be found at http://www.blaxlandweather.com/Images/C ... 017-10.bmp - it's 2.1 MB. You can see that at midnight, the change of MT's day, that the program is carrying over (possibly because the API is still reporting the WD day's rain) and adding the departing day's already recorded rain - as though the heavens had opened.

* Late note: The daily rain figures I quoted were correct at the time but changed after writing this. I delayed posting as I needed to double-check my logic. The yearly figures are correct.

Phew! I hope all that makes sense.

John

[Luc]

Hi John,

The problem here is that WD resets its daily rain values at 09:00.
MT requires a total rain value from 00:00 till 23:59:59, thus a reset at 00:00.

Theoretical the problem can only be solved in WD, which then should calculate two different cumulative rain values.
1. For internal WD the WD program should add rain from 09:00 till 08:59:59 and then reset to zero.
2. For MT the WD program should add rain from 00:00 till 23:59:59 and then reset to zero.

You can ask Brian, but imho I think chances are small he will provide such extra calculations.
In the mean time settings WD's day to start at 00:00 instead of 09:00 is the best work around.

Cheers, Luc

[John B]

Indeed, Luc, you are expressing my own thoughts.

In no way am I criticising Jáchym's system. It's just frustrating that it is not possible to use MT and WD in conjunction in this overlapping of days.

If it works properly during the next rain day which has an overlap of 'days' then, well, I will have to bow to necessity and leave my WD settings as a calendar day and advise my visitors accordingly.

Thanks for reading my waffle, by-the-way.

John

[Luc]

Hi John,
You still have the problem that the historical rain data in MT has wrong values.
This could be solved by an import of the right (rain) data into MT.
I don't think such data will be available because WD had saved the rain values from 09:00 till 09:00 I guess, so an export of these data wouldn't change anything.
Luc

[John B]

Well, it wouldn't change historical records, which is a problem with Jáchym's system of cumulative database information. I could stop the API for a few minutes and then change figures by entering negative data? Perhaps it will at least bring the current year and month figures back to what they should be. I shall try, and see what happens. There's nothing to loose as it's in a mess anyway.

[Luc]

I'm not 100% sure, but I believe the rain calculations are done on the daily maximum values.

Any inserted negative values wouldn't change anything.
Changing one maximum value a day would also change nothing, because there are many of these maximum values per day in a cumulative system.

[Luc]

Hi John,
Just thinking...
When we would execute a mysql query for MT's data base which will set all rain values to zero which has time stamps between 09:00:00 and 23:59:59, the month and year values will come close to those of WD.
Luc

[John B]

I shall think about that, Luc.

Here it is getting on to evening. I have had two small scotches.

The negative entry did not work, by-the-way, as you suspected.

Tomorrow can wait. I am flying at the moment from Broken Hill to Orange.

[Jachym]

Hi,
Luc is right, rain is taken as daily max value (it is much easier to handle in the sql query and daily max should be equivalent to last value). Negative values would not solve the problem and cause serious issues in other scripts because the API would never let negative values so I assumed such values would never be in the db and it would throw fatal errors on several pages.

If I understand this correctly, you need to talk to Brian about this and ask him what exactly is WD sending. For MT it should always be 0-23:59

[John B]

Thanks, both of you, for confirming my analysis.

No, Jáchym, I won't go bothering Brian. I suspect that most of the visitors to the original site could not care less.

The change to the original site seems to have worked (we had a very small amount of rain today and yesterday) so that confirms the situation.

Now, is anyone capable of writing a script that can be used to instigate Luc's idea of removing all rain data recorded in the alldata table from after 09:00 and before the last entry before midnight? When it comes to database manipulation I am less than a novice - I am totally ignorant. Please?

John

[nikokara - BANNED]

Hi John

Open a PhpMyAdmin and make this query

Code: Select all

SELECT * 
FROM alldata
WHERE DateTime
BETWEEN '2017-10-20 9:00'
AND '2017-10-21 8:59:59'

Is the result that what you wand?

[Luc]

We need to delete this period from all dates (since 2013), so the query is not that simple.

[Luc]

Sorry, I should have said:
We have to set rain (R) and rain rate (RR) to zero for all dates with times between 09:00:00 and 23:59:59.
The other weater data in those periods should not be altered.

[nikokara - BANNED]

Hi Luc

I don't understand this..

If the meteorological day in Australia begins and ends at 9 am, why only the rain and rainrate?
What with wind, temp, pressure etc

I mean John has rain readings from 9:00 to 9:00 and the other data from 0:00 to 23:59?

[Jachym]

Most likely everything is like this but the other ones dont matter. The problem with the rain is that the values shown in MT are wrong. Whereas with T, it shows the real avg between 0 and 23:59

[Luc]

The rain values are the only ones which are wrong, because WD maximum dayrain value could differ from MT max dayrain value at 23:55:00.
We are interested for this work around in WD day rain values, so any rain of the next day should not be taken in account. Thats why we reset the values from 09:00-23:55:00 to zero.
Changing RR is not nesessary, because individual rain values (and RR values) wouldn't have any meaning.
As long as the day totals are ok.
We introduce a statistical error because the rain fallen between 09:00-23:59:59 yesterday is considered as the rain of 'today', but the long term totals would be fine.

[John B]

For nikokara, and other interested readers, by way of explanation. If you are already aware of this, please excuse me.

Weather Display is a stand alone program which does not rely on a database. It does however allow data to be transferred to a database. Thereafter it is up to the site developer to do the coding for access to that database and use it as desired.

Meteotemplate relies entirely on its database table (alldata).

Weather Display reports rain information on the basis of adding the last reported reading to that which went before - to daily, monthly and yearly records. The program also allows the user (that's me) to alter those figures to a small degree. I can in a matter of a few moments amend rain readings for today, yesterday, this month and this year (or all at once). This facility is useful if small adjustments are needed and those changes are done in a timely manner.

As you have probably noticed Meteotemplate does not do that. If you've looked at the database itself you will have seen that it does not include rainfall totals - only accumulated rainfall for the day (see the image below). So, if you request some information about rainfall at a particular date, year, month etc., Meteotemplate accesses the period of interest and adds up all the data as reported at midnight (i.e. the reported figure at 23:55).

OK, that's the basics. Now to the practicalities.

The information provided to Metotemplate via the Weather Display program API has been out of synchronisation with the alldata table. That part of the problem has been fixed by changing the Weather Display day from 0900 to midnight. The task at hand is to sort out the mess that is now the historical data in the alldata table.

As Jáchym has mentioned, the other readings are not of importance. If a maximum wind gust is reported on the 'wrong part of a day' it is of no matter in the greater scheme of things.

How can this be done?

Firstly, it's a one-off job and has to be correct. Yes a backup will be an advantage (read, necessary) but a one-time job would be good.

Luc, at this time I am trying to think through your idea as I'm unsure as to its accuracy. My mind is cluttered with ideas, most of which are wrong, but I need a bit of time to sort out all of them. I'm not saying you are wrong, just that my logic seems to be rebelling for some reason. Perhaps tomorrow will bring clarity.

[nikokara - BANNED]

John

Do this sql query

Code: Select all

SELECT * 
FROM `alldata` 
WHERE `R` >0

and tell me how many rows you have

and then a second query

Code: Select all

SELECT * 
FROM `alldata` 
WHERE `R` >0 and  `RR` >0

and tell me again the number off rows

[John B]

That's going to be a large number, my friend.

What is the reasoning?

[nikokara - BANNED]

I dont wand the data, the number off rows-lines only, it takes max 5 second
I will see if i can export the data, then edit, and then reinput in the database, i have do some tests but i have no many data for rain.

[weather display]

WD does have 2 internal rain totals
i.e midnight to midnight always, regardless if 9am reset used
so I could send that data even if 9am reset used in WD?

NZ uses 9am reset, but I myself have chosen to use midnight for a while now..it does make more sense
for australia,9am (EST) is the midnight greenwich mean time (UTC), so it probably relates to when they were sending data back to the UK orginally

[Luc]

WD does have 2 internal rain totals
i.e midnight to midnight always, regardless if 9am reset used
so I could send that data even if 9am reset used in WD

Hi Brian,
Yes, that would be a good solution. Then one has still the choice to have both the 9am reset in WD and the midnight reset for MT.
Thanks.
Cheers, Luc