Help importing history data

Post Reply
yds
Newbie
Newbie
Posts: 2
Joined: Mon Jan 22, 2018 1:14 am
Station model: Davis VP2
Software: WeatherDisplay

Help importing history data

Post by yds » Mon Jan 22, 2018 12:15 pm

Just installed Jachym's excellent templates. Need to import data from my WeatherDisplay install. I have data going back to June 2007 I would like to import. I've read the wiki, and if I understand correctly, I can use the WD api to both provide the ongoing updates, and to import the historical data.

From the WD control panel, I can bring up the Meteotemplate setup screen. Can someone help explain how I use it to import all my historical data? Do I have to do it one month at a time? Below is the screen shot from the WD api page. I'm assuming I use the "All Data Log File Export" section at the bottom to do this. It's not clear to me how I'm supposed to use the "Month" and "Year" boxes. And do I use both the "create ...." buttons, and then the "Upload" buttons?

Any assistance appreciated.

Capture.PNG
Capture.PNG (13.8 KiB) Viewed 6739 times

User avatar
jay_hoehn
Language Admin
Language Admin
Posts: 132
Joined: Sun Aug 20, 2017 1:45 pm
Location: Las Cruces, NM USA
Station model: Vantage Pro 2 Plus
Software: WeeWx
Contact:

Re: Help importing history data

Post by jay_hoehn » Mon Jan 22, 2018 5:22 pm

yds,

I used Weather in the past and if I remember correctly you need to import one month at a time. The title of the import box is not clear. It is not all data but import to the alldata table. I also could not get the upload to work. I took the files from the computer, that were created when you press the create button, in the weather display folder uploaded them with my ftp then imported them to the database from there using the control panel in Meteotemplate.

Hope that helps,

Jay

yds
Newbie
Newbie
Posts: 2
Joined: Mon Jan 22, 2018 1:14 am
Station model: Davis VP2
Software: WeatherDisplay

Re: Help importing history data

Post by yds » Tue Jan 23, 2018 12:34 am

Wow. One month at a time. So for me, that's 132 months. Times two, since I also have extra sensors. Then manually import one at a time. Thanks for the info Jay. Need to decide if I want to move all that data then.

User avatar
Ken Good
Advisor
Advisor
Posts: 62
Joined: Thu Dec 28, 2017 4:10 am
Location: Tyler, Texas
Station model: Davis Vantage Pro 2 Plus
Software: Meteobridge
Contact:

Re: Help importing history data

Post by Ken Good » Tue Jan 23, 2018 1:46 am

If your data is on Weather Underground you can download it in bulk. I did about 6 months at a time.

Ken
Image

tpweather
Observer
Observer
Posts: 25
Joined: Sat Apr 07, 2018 12:52 pm
Location: Tom Price WA
Station model: Davis Vantage Pro 2
Software: Weather Display
Contact:

Re: Help importing history data from WD

Post by tpweather » Wed Apr 11, 2018 3:20 pm

Hi All
I am having some problems with importing data. I have captured the data files through WD using the metotemplate function in control panel with no problem. I have uploaded them to my server and I am trying to import them using database import function within the template.
I think, I have tried all settings though obviously not as I keep getting errors.
Here is the beginning of one of my files trying to import:
YYYY/m/d H:i,avtemperature(C),avhumidity,avbarometer(hpa),avwindspeed(km/h),gustspeed(km/h),rainfall(mm),rainrate(mm/hr),avdirection,avsolar
2015/10/01 00:00,26.1,4.0,1021.3,0.0,3.7,0.0,0.0,68,0.0
2015/10/01 00:05,26.1,4.0,1021.2,0.0,3.7,0.0,0.0,68,0.0
2015/10/01 00:10,26.0,4.0,1021.1,0.0,1.9,0.0,0.0,68,0.0
2015/10/01 00:15,25.9,4.3,1021.1,0.0,1.9,0.0,0.0,68,0.0

No matter what date and time format I put in including the fields I get this:
Parsing line 0 finished.
Checking validity of data based on your limits in Main settings...
Checking date and time...
There seems to be a problem with the date and time. Either it has not been parsed correctly or it is a date/time in the future! Check the format you specified and also the actual values.
Minimum temperature allowed: -100 C
Maximum temperature allowed: 100 C
Temperature is outside the allowed limits! Temperature value will be ignored!
Minimum humidity allowed: 0 %
Maximum humidity allowed: 100 %
Humidity OK.
Minimum pressure allowed: 900 hpa
Maximum pressure allowed: 1100 hpa
Pressure is outside the allowed limits! Pressure value will be ignored!

So apart from the time format setup I also have a problem with the settings I put in the setup file.

I'm pretty sure it's staring me in the face. :oops: Can anyone please advise or guide me.
Many thanks
Chris
Image

User avatar
jay_hoehn
Language Admin
Language Admin
Posts: 132
Joined: Sun Aug 20, 2017 1:45 pm
Location: Las Cruces, NM USA
Station model: Vantage Pro 2 Plus
Software: WeeWx
Contact:

Re: Help importing history data

Post by jay_hoehn » Wed Apr 11, 2018 9:54 pm

Chris,

Is it possible that WD is giving you YYYY/d/m instead of YYYY/m/d?

Jay

tpweather
Observer
Observer
Posts: 25
Joined: Sat Apr 07, 2018 12:52 pm
Location: Tom Price WA
Station model: Davis Vantage Pro 2
Software: Weather Display
Contact:

Re: Help importing history data

Post by tpweather » Fri Apr 13, 2018 11:24 am

Hi Jay
Thanks for the reply. I have checked that. Not the issue. Attached is one of the files WD creates.
Cheers for the help
Attachments
meteotemplatelog12018.zip
(63.41 KiB) Downloaded 125 times
Image

tpweather
Observer
Observer
Posts: 25
Joined: Sat Apr 07, 2018 12:52 pm
Location: Tom Price WA
Station model: Davis Vantage Pro 2
Software: Weather Display
Contact:

Re: Help importing history data

Post by tpweather » Fri Apr 13, 2018 1:14 pm

I think I have all the settings correct now though hard to tell as I get this error now:
Forbidden
You don't have permission to access /site/admin/testImportFile.php on this server.
Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.
I set it to 777 but same issue. Any ideas anyone?
Image

tpweather
Observer
Observer
Posts: 25
Joined: Sat Apr 07, 2018 12:52 pm
Location: Tom Price WA
Station model: Davis Vantage Pro 2
Software: Weather Display
Contact:

Re: Help importing history data

Post by tpweather » Sun Apr 15, 2018 3:38 pm

Turned out to be a modsec rule. All sorted now
Image

User avatar
jay_hoehn
Language Admin
Language Admin
Posts: 132
Joined: Sun Aug 20, 2017 1:45 pm
Location: Las Cruces, NM USA
Station model: Vantage Pro 2 Plus
Software: WeeWx
Contact:

Re: Help importing history data

Post by jay_hoehn » Mon Apr 16, 2018 12:32 am

Chris,

Glad you got it sorted out.

Jay

User avatar
aznetcowboy
Advisor
Advisor
Posts: 50
Joined: Wed Apr 04, 2018 6:21 pm
Location: Tucson, AZ
Station model: Davis Vantage Pro2
Software: Cumulus
Contact:

Re: Help importing history data

Post by aznetcowboy » Thu Apr 19, 2018 7:00 pm

All this is very interesting, and useful. However I am using Cumulus for my software. I can import the current readings with no problems, but I don't see how to import historical data from my Cumulus database. Has any done a Cumulus historical database import into meteotemplate? :?:
Tom Wills
Kingston Knolls Terrace, Tucson, AZ 85710

El Gheko Neighborhood Weather Station: http://www.elgheko.us
El Gheko meteoTemplate Weather Station: http://www.elgheko.us/mtp/
El Gheko Neighborhood Association: http://na.elgheko.us
Certified Skywarn Trained Observer #1569
Image

Post Reply