Problems with data import

[packman2008]

I'm trying to import a lot of old data into the Template (v18) but I'm having problems with a date parsing error, i.e. when I try a test import I get...

Parsing line 1 finished.
Checking validity of data based on your limits in Main settings...
Checking date and time...
There seems to be a problem with the date and time. Either it has not been parsed correctly or it is a date/time in the future! Check the format you specified and also the actual values.

The first two lines of my data file are...

Date,Time,temp,pressure,humidity,wind_speed,wind_deg,rain_1h,snow_1h,Daily Rain
1979-01-01,00:00,-5.9,1015,85,1,40,,,0

and when I show field numbers I get the following which looks OK.

Loading CSV/text file from https://blackpool.weather-watch.com/data/part.csv

File loaded successfully.
Header row in the file, skipping first line, using for field labels.

Starting to parse data....

The tested file has 17 data sets.
Field delimiter is comma (,)
First dataset is: 1979-01-01,00:00,-5.9,1015,85,1,40,,,0
The header row contains 10 columns.
The data row contains 10 fields.
Loaded data:
Field Field Number Value (first data set)
Date 0 1979-01-01
Time 1 00:00
temp 2 -5.9
pressure 3 1015
humidity 4 85
wind_speed 5 1
wind_deg 6 40
rain_1h 7 null
snow_1h 8 null
Daily Rain 9 0

I've set the date/time as being in two fields: 0=Date as 'Y-m-d' and 1=Time as 'H:i'.

I think I have everything correct and all the data is definitely in the past (the 17 lines of test data are all from 1979).

Can anyone suggest what I'm doing wrong?

Thanks,
Chris

[davidefa]

It should be ok. There's a check that the import date is greater than 1980-01-01 ( your is not ), and the warning is misleading.
I modified the limit to 1970, uncompress the files in the admin folder.
These should be the import configuration ( you need to verify if it is correct )

imp1.png (167.86 KiB) Viewed 6449 times

imp2.png (124.68 KiB) Viewed 6449 times

[davidefa]

imp3.png (144.11 KiB) Viewed 6449 times

[packman2008]

Thanks for that

I'd just discovered the 1980 minimum date before lunch and had successfully tested by changing the dates to 1980-02-02. Oddly setting the date to 1980-01-01 didn't work, but maybe the test is '<' and not '<='. I was wondering over lunch time if changing the minimum date to 1970-01-01 would cause any other problems. I know MySQL is happy with dates between 1000-01-01 and 9999-12-31 so it shouldn't cause MySQL a problem but I didn't know if there was something in the Template that might get upset with earlier dates.

[packman2008]

360,000-ish rows imported. Thanks for the fix.

[QuilpieGuy]

Hi packman2008

How long did the import take??
I'm importing approx 450,000 rows, I've been looking at a screen (OK not continuously ) for about three hours that's showing

Importing data from CSV/text file...

Errors will be highlighted in red...

Loading CSV/text file from <file URL removed>

I did a "test import" on a subset of 10 records, and that flagged no errors.

Just wondering how patient I need to be .

Regards

[packman2008]

10-15 minutes if I remember correctly. Definitely not hours. You should see the import data scrolling up the screen so you can guage how fast it's going.

[QuilpieGuy]

Thanks for your quick response.
So somethings isn't working, There is no that scrolling at all, just that static screen I mentioned.
Guess I'll just try and do it via a direct SQL insert statement.