Import data issue (timeout)

[i_fiorentino]

Hi,
i would insert weather data from Weather Display logfiles into meteotemplate db.
Every WD logfiles contains 44640 rows (1*60*24*31).

I've tried to use meteotemplate import data menu but this process ends after only 3 days (4320 rows).
What can i do to import the entire logfiles?
How many minutes use to import all month?

Thank you so much for any help.
Regards,


Alessandro

[davidefa]

Importing 44K lines should not take too long ( are you saying you are hitting some timeout ).
Post a couple of data files so we can give it a try.

[i_fiorentino]

Importing 44K lines should not take too long ( are you saying you are hitting some timeout ).
Post a couple of data files so we can give it a try.

Sure,
here it is: https://www.dropbox.com/s/1elazsjuzox14 ... g.txt?dl=0
https://www.dropbox.com/s/7yug2kerfz2a7 ... g.txt?dl=0
Thanks in advance,

Alessandro

[davidefa]

I imported correctly both files using following settings:
If this does not work for you we need to understand why the script times out ( or use phpmyadmin for the import process, a conversion of the data is required )

import01.png (136.06 KiB) Viewed 3085 times

import02.png (85.43 KiB) Viewed 3085 times

import03.png (88.14 KiB) Viewed 3085 times

[i_fiorentino]

Thank you so much for your reply and support.
Strange.

My settings are the same except Field delimiter.
In my case is set to "tab" while your is "space"

With space i've a lot of errors because month values goes empty.
Same stop after three days of imported data

Regards,


Alessandro

[i_fiorentino]

Finally i've imported all files data.
I've three issues now.

1) Wind gust record doesn't imported......i don't know why
2) I've executed reformat.php script but it seems no working....my db remains with 1 minute interval.
3) Live data recordered correctly every 5 minutes but it creates three rows with the same values:


How to fix this bug?

Thanks in advance for any help.
Regards,


Alessandro

[davidefa]

Hi Alessandro, regarding point 3) you are calling api.php every 10 seconds, I think you should change to 60 seconds.
I think ( not sure about this ) that your api.php takes very long to execute and this results in multiple instances executed in the same time, each saving to the database ( you could check this in phpmyadmin ).

Regarding point 2) I executed the script reformat.php and it works correctly ( but I had only very limited data in db )
The script is quite simple you could try executing in phpmyadmin:

Code: Select all

CREATE TABLE new SELECT DateTime, avg(T), max(Tmax), min(Tmin), avg(H), avg(D), avg(W),max(G),B,avg(RR),max(R),avg(P),max(S),avg(A) FROM alldata GROUP BY round(UNIX_TIMESTAMP(DateTime) DIV 300)

This should result in a new database table called 'new' with a record every 5 minutes

[i_fiorentino]

Ok,
i've renamed new table in alldata table and i've fixed db issue!
Thank you so much Davide!

Now, i would edit wind gust data because during import data i've lost my alltime record.
I've edited wind gust data row with a new one and save it but i've seen the new data doens't show on my home page.
What's wrong?

Thank you so much.
Regards,


Alessandro

[davidefa]

Now, i would edit wind gust data because during import data i've lost my alltime record.
I've edited wind gust data row with a new one and save it but i've seen the new data doens't show on my home page.
What's wrong?

Thank you so much.
Regards,


Alessandro

I've noticed that too, sometimes the database update does not work ( but it does not return any error )
You can always use phpmyadmin to edit a row

phpmyadmin.png (201.47 KiB) Viewed 2700 times

[i_fiorentino]

Thanks for reply!
Unfortunately i've not "edit" field on every db row.


How have you done?

Ty,


Alessandro

[davidefa]

I think you have to setup these things: ( particularly set DateTime as primary key ) type, null and default columns as shown

dbsetup.png (199 KiB) Viewed 2596 times