Help importing history data
-
- Newbie
- Posts: 2
- Joined: Mon Jan 22, 2018 1:14 am
- Station model: Davis VP2
- Software: WeatherDisplay
Help importing history data
Just installed Jachym's excellent templates. Need to import data from my WeatherDisplay install. I have data going back to June 2007 I would like to import. I've read the wiki, and if I understand correctly, I can use the WD api to both provide the ongoing updates, and to import the historical data.
From the WD control panel, I can bring up the Meteotemplate setup screen. Can someone help explain how I use it to import all my historical data? Do I have to do it one month at a time? Below is the screen shot from the WD api page. I'm assuming I use the "All Data Log File Export" section at the bottom to do this. It's not clear to me how I'm supposed to use the "Month" and "Year" boxes. And do I use both the "create ...." buttons, and then the "Upload" buttons?
Any assistance appreciated.
From the WD control panel, I can bring up the Meteotemplate setup screen. Can someone help explain how I use it to import all my historical data? Do I have to do it one month at a time? Below is the screen shot from the WD api page. I'm assuming I use the "All Data Log File Export" section at the bottom to do this. It's not clear to me how I'm supposed to use the "Month" and "Year" boxes. And do I use both the "create ...." buttons, and then the "Upload" buttons?
Any assistance appreciated.
- jay_hoehn
- Language Admin
- Posts: 135
- Joined: Sun Aug 20, 2017 1:45 pm
- Location: Las Cruces, NM USA
- Station model: Vantage Pro 2 Plus
- Software: WeeWx
- Contact:
Re: Help importing history data
yds,
I used Weather in the past and if I remember correctly you need to import one month at a time. The title of the import box is not clear. It is not all data but import to the alldata table. I also could not get the upload to work. I took the files from the computer, that were created when you press the create button, in the weather display folder uploaded them with my ftp then imported them to the database from there using the control panel in Meteotemplate.
Hope that helps,
Jay
I used Weather in the past and if I remember correctly you need to import one month at a time. The title of the import box is not clear. It is not all data but import to the alldata table. I also could not get the upload to work. I took the files from the computer, that were created when you press the create button, in the weather display folder uploaded them with my ftp then imported them to the database from there using the control panel in Meteotemplate.
Hope that helps,
Jay
-
- Newbie
- Posts: 2
- Joined: Mon Jan 22, 2018 1:14 am
- Station model: Davis VP2
- Software: WeatherDisplay
Re: Help importing history data
Wow. One month at a time. So for me, that's 132 months. Times two, since I also have extra sensors. Then manually import one at a time. Thanks for the info Jay. Need to decide if I want to move all that data then.
- Ken Good
- Advisor
- Posts: 62
- Joined: Thu Dec 28, 2017 4:10 am
- Location: Tyler, Texas
- Station model: Davis Vantage Pro 2 Plus
- Software: Meteobridge
- Contact:
Re: Help importing history data
If your data is on Weather Underground you can download it in bulk. I did about 6 months at a time.
Ken
Ken
-
- Observer
- Posts: 25
- Joined: Sat Apr 07, 2018 12:52 pm
- Location: Tom Price WA
- Station model: Davis Vantage Pro 2
- Software: Weather Display
- Contact:
Re: Help importing history data from WD
Hi All
I am having some problems with importing data. I have captured the data files through WD using the metotemplate function in control panel with no problem. I have uploaded them to my server and I am trying to import them using database import function within the template.
I think, I have tried all settings though obviously not as I keep getting errors.
Here is the beginning of one of my files trying to import:
YYYY/m/d H:i,avtemperature(C),avhumidity,avbarometer(hpa),avwindspeed(km/h),gustspeed(km/h),rainfall(mm),rainrate(mm/hr),avdirection,avsolar
2015/10/01 00:00,26.1,4.0,1021.3,0.0,3.7,0.0,0.0,68,0.0
2015/10/01 00:05,26.1,4.0,1021.2,0.0,3.7,0.0,0.0,68,0.0
2015/10/01 00:10,26.0,4.0,1021.1,0.0,1.9,0.0,0.0,68,0.0
2015/10/01 00:15,25.9,4.3,1021.1,0.0,1.9,0.0,0.0,68,0.0
No matter what date and time format I put in including the fields I get this:
Parsing line 0 finished.
Checking validity of data based on your limits in Main settings...
Checking date and time...
There seems to be a problem with the date and time. Either it has not been parsed correctly or it is a date/time in the future! Check the format you specified and also the actual values.
Minimum temperature allowed: -100 C
Maximum temperature allowed: 100 C
Temperature is outside the allowed limits! Temperature value will be ignored!
Minimum humidity allowed: 0 %
Maximum humidity allowed: 100 %
Humidity OK.
Minimum pressure allowed: 900 hpa
Maximum pressure allowed: 1100 hpa
Pressure is outside the allowed limits! Pressure value will be ignored!
So apart from the time format setup I also have a problem with the settings I put in the setup file.
I'm pretty sure it's staring me in the face. Can anyone please advise or guide me.
Many thanks
Chris
I am having some problems with importing data. I have captured the data files through WD using the metotemplate function in control panel with no problem. I have uploaded them to my server and I am trying to import them using database import function within the template.
I think, I have tried all settings though obviously not as I keep getting errors.
Here is the beginning of one of my files trying to import:
YYYY/m/d H:i,avtemperature(C),avhumidity,avbarometer(hpa),avwindspeed(km/h),gustspeed(km/h),rainfall(mm),rainrate(mm/hr),avdirection,avsolar
2015/10/01 00:00,26.1,4.0,1021.3,0.0,3.7,0.0,0.0,68,0.0
2015/10/01 00:05,26.1,4.0,1021.2,0.0,3.7,0.0,0.0,68,0.0
2015/10/01 00:10,26.0,4.0,1021.1,0.0,1.9,0.0,0.0,68,0.0
2015/10/01 00:15,25.9,4.3,1021.1,0.0,1.9,0.0,0.0,68,0.0
No matter what date and time format I put in including the fields I get this:
Parsing line 0 finished.
Checking validity of data based on your limits in Main settings...
Checking date and time...
There seems to be a problem with the date and time. Either it has not been parsed correctly or it is a date/time in the future! Check the format you specified and also the actual values.
Minimum temperature allowed: -100 C
Maximum temperature allowed: 100 C
Temperature is outside the allowed limits! Temperature value will be ignored!
Minimum humidity allowed: 0 %
Maximum humidity allowed: 100 %
Humidity OK.
Minimum pressure allowed: 900 hpa
Maximum pressure allowed: 1100 hpa
Pressure is outside the allowed limits! Pressure value will be ignored!
So apart from the time format setup I also have a problem with the settings I put in the setup file.
I'm pretty sure it's staring me in the face. Can anyone please advise or guide me.
Many thanks
Chris
- jay_hoehn
- Language Admin
- Posts: 135
- Joined: Sun Aug 20, 2017 1:45 pm
- Location: Las Cruces, NM USA
- Station model: Vantage Pro 2 Plus
- Software: WeeWx
- Contact:
Re: Help importing history data
Chris,
Is it possible that WD is giving you YYYY/d/m instead of YYYY/m/d?
Jay
Is it possible that WD is giving you YYYY/d/m instead of YYYY/m/d?
Jay
-
- Observer
- Posts: 25
- Joined: Sat Apr 07, 2018 12:52 pm
- Location: Tom Price WA
- Station model: Davis Vantage Pro 2
- Software: Weather Display
- Contact:
Re: Help importing history data
Hi Jay
Thanks for the reply. I have checked that. Not the issue. Attached is one of the files WD creates.
Cheers for the help
Thanks for the reply. I have checked that. Not the issue. Attached is one of the files WD creates.
Cheers for the help
- Attachments
-
- meteotemplatelog12018.zip
- (63.41 KiB) Downloaded 184 times
-
- Observer
- Posts: 25
- Joined: Sat Apr 07, 2018 12:52 pm
- Location: Tom Price WA
- Station model: Davis Vantage Pro 2
- Software: Weather Display
- Contact:
Re: Help importing history data
I think I have all the settings correct now though hard to tell as I get this error now:
I set it to 777 but same issue. Any ideas anyone?Forbidden
You don't have permission to access /site/admin/testImportFile.php on this server.
Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.
-
- Observer
- Posts: 25
- Joined: Sat Apr 07, 2018 12:52 pm
- Location: Tom Price WA
- Station model: Davis Vantage Pro 2
- Software: Weather Display
- Contact:
Re: Help importing history data
Turned out to be a modsec rule. All sorted now
- jay_hoehn
- Language Admin
- Posts: 135
- Joined: Sun Aug 20, 2017 1:45 pm
- Location: Las Cruces, NM USA
- Station model: Vantage Pro 2 Plus
- Software: WeeWx
- Contact:
Re: Help importing history data
Chris,
Glad you got it sorted out.
Jay
Glad you got it sorted out.
Jay
- aznetcowboy
- Advisor
- Posts: 50
- Joined: Wed Apr 04, 2018 6:21 pm
- Location: Tucson, AZ
- Station model: Davis Vantage Pro2
- Software: Cumulus
- Contact:
Re: Help importing history data
All this is very interesting, and useful. However I am using Cumulus for my software. I can import the current readings with no problems, but I don't see how to import historical data from my Cumulus database. Has any done a Cumulus historical database import into meteotemplate?
Tom Wills
Kingston Knolls Terrace, Tucson, AZ 85710
El Gheko Neighborhood Weather Station: http://www.elgheko.us
El Gheko meteoTemplate Weather Station: http://www.elgheko.us/mtp/
El Gheko Neighborhood Association: http://na.elgheko.us
Certified Skywarn Trained Observer #1569
Kingston Knolls Terrace, Tucson, AZ 85710
El Gheko Neighborhood Weather Station: http://www.elgheko.us
El Gheko meteoTemplate Weather Station: http://www.elgheko.us/mtp/
El Gheko Neighborhood Association: http://na.elgheko.us
Certified Skywarn Trained Observer #1569