Data Import from Weather Display (logfiles)

[Juan Maria Vilaró]

Hello, I request help to import historical data.
I am using Weather display, and I have data from 1/1/2010, I have a WMR200 station and as a server I use "Xammp iy Xnet" the data uploads correctly in the template.
I do the import steps, the "logfiles" are read correctly and before importing I do a test, which confirms that it is OK, then I do the second step, which is to import the file, but it does not read the entire month, maximum 6 or 7 days and of course the data is not uploaded to the "alldata" database, can someone help me or know what the problem could be? thank you so much.

[FSC830]

Is the actual data from Weather station added to the alldata table?
In which format is the historical data?

If the data is available in csv format you can easily add it to the database.
Mandatory is that the order of fields (values) is matching the alldata table structure.

Regards

[Juan Maria Vilaró]

Yes, the template is correctly updated with the real data, the format of the WD "logfiles" since 2010 we have them in ".txt"
When I do the test, the import program recognizes the lines and the data, responds with OK, but it does not import them into the database, I do not have them in CSV format.
Thanks for your reply

[FSC830]

I did not use the import program until today, I directly added the data in mysql database using mysql commands.
But my data was in .csv format.

Do you have SSH access to your database host and can run mysql commands?
If so, can you attach an example of the .txt format?

Regards

[Juan Maria Vilaró]

Hello, good morning, valued collaborator, it is fair to explain things as they are, I particularly have no problems with my Meteotemplate template and everything works correctly, I have a very competent and helpful server, the reality is that I am trying to help a friend that unlike me, its server is "localhost" it uses Xnet to upload the data from Weather Display and Xnet mysql to feed the "data" database and the "alldata" table of meteotemplate, this function correctly updates the template from the year 2020 until today, the problem has arisen when you try to upload your history, I have tried to help you but I get your data to upload to the database despite the fact that no error is detected, I attach a logfile of 12010lg.txt
Thank you very much for your great interest in helping.
Greetings

[FSC830]

Hi,
today I have had some time left and tested the import of your file.
Long story short: importing was successfull, all 44.175 lines have been imported without an issue.
In addition I tried to import the data directly in mysql, which also succeeds with some minor changes in advance.

I did wrote a very short shell script which converts the data from Weather Display into Meteotemplate format (i.e. merge the date/time fields, sort out the fields not used by MT and bring data in an order for direct importing into alldata table).

This works too and at the end I only need to delete a single line with a timestamp 0000-00-00 00:00
Afterwards I dropped the table, created it new and used to import page from MT. After entering the field order the import was done in less than a minute.

Screenshots from testing (I renamed the file to xxx.txt).

Import_01.png (269.01 KiB) Viewed 357 times

Import_02.png (26.13 KiB) Viewed 357 times

Import_03.png (107.45 KiB) Viewed 357 times

Regards

[FSC830]

Here is the result in mysql (cant add a 4th picture in previous post)?

import_CLI_02.png (25.5 KiB) Viewed 356 times

Regards

[Juan Maria Vilaró]

Thank you very much, I'll try tomorrow, what I don't know is the MYSQL import, and the shell script, taking into account that the person I'm trying to help uses Xammp, how can I do it to import directly, I don't know if it's too much trouble, but it will help I would be extremely grateful if you could teach me how to do it.
Again, thank you very much.
Juan

[FSC830]

Hi,

attached is an archive with a description and the shell script I used (script is may be a bit oversized, it contains only a few lines) .

Be sure that you (and/or the person you tried to help) understand all tasks to avoid any data loss.
Actually I am not very familiar with MariaDB, but as far as I found in Google the import command should be the same. The databases are mainly identical.

If you have the possibility to check all tasks in a test environment this is strongly recommended.
The script runs at my VM about 5-10 minutes to convert the abt. 44.000 records.
If you are using larger files, the time will increase, so dont be worry about that.

Regards

import_data.zip

(127.23 KiB) Downloaded 29 times

[Juan Maria Vilaró]

Dear collaborator, I am very grateful for the interest shown in guiding me, unfortunately my knowledge is somewhat limited in this field, even after having read and translated the "PDF" I am not able to correctly carry out the instructions received, in any case many Thank you for your invaluable help.
Juan

[FSC830]

Hi Juan,
sad to hear that the PDF is not helping you. .
At which point exactly do you stuck?

Regards