How to import lost data

[magical46]

Hi, recently my web site seemed to have disappeared, on contacting my host they have now sorted the issue and moved everything to new servers, unfortunately in the process they seem to have cleared all the data from my "MySQL" data base, that's 10 years worth of data gone. I have all the files in Weather display, but having looked at the data import page on my site I am totally confused (being a bit of a dinosaur in these things). How do I transfer the data from my PC to my web site, which files do I upload and to where?
Because of the change of server my FTP now points to the wrong place and I am waiting to hear from them what the new details for FTP should be as they haven't provided me with the format to use for the FTP.
They did however manage to bill me for the website!!!
Phil

[davidefa]

You can post your data ( at leat part of it ) in csv format, add a pair of rows:
- first with the name of the fields
- second with the unit of measure of the fields

[magical46]

Hi, thanks for your reply. Unfortunately , being a dinosaur , I didn't really understand it. Since the host moved my site to new servers I have had to reset my FTP file in Weather display, and I then found that all the details (user name , host etc) for the MySql database had changed so I have now amended them in the template main setup page, done the "test" which has set up the database and my site is now updating as it should.
What I now need to do is get the historic data from Weather display on my pc to my site. I know there were clear instructions on how to do this when I first started using meteotemplate but that was way back in a very early version, since then a lot has changed, so I need to know which files to import from weather display (webfiles I assume), but looking at the data import section of the template just completely baffles me.!!
Phil

[davidefa]

Meteotemplate can import data in csv format ( is a simple text format for data exchange ).
The steps are as follow:
- first you need to export data from Weather display in csv format ( data is divided in rows and columns )
- second you have to import this data in meteotemplate
Unfortunately the import phase is not 'fully automatic', but is quite simple conceptually:
for each field you want to import ( for example Temperature,Humidity... ) you have to specify in which column it resides and the unit of measure
Easier to do than to describe.
Sometime the import process fail at some point ( some 'strange'/unexpected data ) so a minimum of manual intervention may be required ( nothing impossible )

import.png (115.22 KiB) Viewed 14847 times

[magical46]

I have created a sub directory in the "install" directory called dbupload. I have uploaded a few log files to that directory (they are .txt files). if I go to http://www.philsweather.co.uk/meteotemp ... 200804.txt then the page displays showing the text data for that file, if I the copy and paste that url into the database upload page and click "show field numbers it says it cannot find that page. Where am I going wrong?
Phil

[davidefa]

1) the file should be placed inside the admin folder, or a subdirectory ( and not the install folder as stated in the import page )
2) the settings are as shown in the attached images
3) select the correct units ( for example wind speed is expressed in Km/h, mph... )
4) now 'Show fields number' and 'Test import' should work as expected
With this settings data import correctly... but I'm not shure data are correct:
- for example on 2008-04-30 21:58 yearly rain pass from 14.7 to 129 in 1 minute
- for example on 2008-04-24 12:32 ( first row ) yearly rain is less than monthly rain
- daily rain never resets
nonetheless data are correctly shown ( at first glance )

import1.png (182.83 KiB) Viewed 14834 times

import2.png (132.07 KiB) Viewed 14834 times

import3.png (161.26 KiB) Viewed 14834 times

[davidefa]

graph1.png (269.63 KiB) Viewed 14834 times

graph2.png (374.07 KiB) Viewed 14834 times

[magical46]

Many thanks for that. The rain data maybe odd due to the fact it was the first I had in WD so not a full month/year of stuff in WD.
is the data_import_200804 the full path required if I move it to the a sub directory in the admin directory?
I have all the data from 200804 to date to import, do I have to do each txt file individually, if so its a good job I have a couple of days off work?!!!
Phil

[davidefa]

is the data_import_200804 the full path required if I move it to the a sub directory in the admin directory?

No the path is relative to the admin directory, so if you put the data in admin/dbupload/200804.txt
simply set URL to dbupload/200804.txt

I have all the data from 200804 to date to import, do I have to do each txt file individually, if so its a good job I have a couple of days off work?!!!

If you have the files separated ( one for each month ) it could be more convenient to join them together

es: if you are on a windows os open a command prompt in the directory where the files reside and type

Code: Select all

type 20*.txt > 2008_2020.txt

Last option is import each file individually... but in this case a cup of coffee 'at hand' may be advisable ( and when I say 'at hand' I mean at each hand ; - )

[magical46]

Thank you so much for all this help, I am now working my way through the files to import (one at a time, lots of coffee!!).
I have one more query. The files I have done so far are the ones I originally uploaded when I first set up meteotemplate which go up to 2016. I seem to remember something about changing the date format as , for example, the file for April 2016 is listed as 201604.txt however in the log files in Weather display would be 42016. Do I need to change this or will they get put in the right fields as we have specified what field the months and years go in?
Phil

[davidefa]

The file name is not important... the important thing is that the file format does not change ( the field order is same as specified )

[magical46]

Many thanks for all the help, I now have all the historic data from 2008 to date in the data base.Today was a pain as kept getting "504 gateway timeout" but got there in the end.
The only slight anomaly is that the data base block is saying it has data from 1999 to date and although the history plugin show all those years, 1999 to 2008 have no data in them. Not sure how it got 1999 as a start year!!
Phil

[davidefa]

Perhaps one ( or more ) data field were ill-formed, and so where imported incorrectly.
You could use the page Interactive Graph and set a Custom interval to 'hunt' these 'outliers'.
If these are a few you can simply delete them ( in the control panel there is a Bulk Database Editing ).
If these are a lot you should also see a corresponding 'hole' somewhere in the graph.
In this case we should investigate it ( maybe the format of a whole month was different... )

P.S.
If you feel brave you can do it from the mysql control panel ( phpmyadmin ) see attached example