Database and Netatmo

[El_Viejo]

Until davidefa solved the problem (thanks, Davide), for 100 days, the users of this weather station have not been able to keep the database updated.

I am trying to create a database with the readings of those 100 days and then export it to the template.
Netatmo allows me to save the data in two types of files: XLSX and CSV.
I have done several tests and I am in favor of the CSV format. In my inexperienced opinion, it seems to me the most compatible with the format used by the template.

But the first doubt that arises is the "Date Time" field. I consult my SQL database with phpmyadmin and I find a difference. In SQL the format is "yyyy-mm-dd hh:mm:ss". In CSV the format is "yyyy-mm-dd hh:mm:mm". That is, the seconds are missing.
If after several hours of work I manage to create a macrotable with the readings of the 100 days, what will be the consequence of the missing seconds in the "Date Time" field? Will it be interpreted as zero or the export of the table will not be successful?

I work with two servers installed on two Raspberry Pi. On one RPi 4, I have successfully running the template since December2019. On a RPi 5, I just installed another Papaya version and here I will test.
I am 82 years old, very inexperienced, but I can decide what to use my free time for. However, I will be very grateful to receive some guidance from the experts this forum.

Thanks in advance.

[davidefa]

Please post your csv file ( yes, csv format is the preferred one ), I'll show you how to configure meteotemplate to import it

[El_Viejo]

Thank you Davide.
I am working on the files downloaded from Netatmo.
But it is 100 days of data and a lot of rows in Excel. Also, Netatmo downloads files for each of the installed modules and updates the readings every 5 minutes.
In the case of the rain gauge, it downloads the "R", but not the "RR", which I have to calculate.
The pressure is measured in an indoor module, which has other sensors that I am not interested in for the database.
It will take me two or three days.

[davidefa]

You can leave all fields in the csv file, in the import process you can select only the fields you want.
Regarding the rain rate, yes you have to calculate it from the rain data ( meteotemplate does not calculate it for you ).
Generally a 'weighted average' is used to calculate the rain rate, to smooth the rain rate, which is generally a 'little jumpy', especially at low rain rates.

P.S.
You forgot to add that you are an old man and 'bla bla'.. ( not bad for an 'old man' ; - )

[El_Viejo]

Due to a set of circumstances, I have not been able to dedicate my time to configure my files of the hundred days without connection to the Netatmo servers.
When working with these files, I am experiencing the following doubts:

1- It has a "Timestamp" column that seems to me to be related to UTC time measurement. That parameter does not appear in the SQL database, can I delete it?

2 - The four Excel/CSV files downloaded from Netatmo servers are slightly different. The start and end times of the measurements are not exactly the same, nor are the number of rows.
The start times range from 00:00:32 to 00:00:39 and similar differences exist for the end of the day.
The number of rows in the files ranges from 8862 to 8872.
For temperature, humidity, pressure and wind data I can unify all data in a single CSV file, with 10 minute intervals, without loss of quality.
But, in the case of the rain gauge, even if to the "R" data, I add the "RR" data manually, I find it very difficult to integrate them into the previous CSV file, without the loss of some very relevant data in the hours of very heavy rainfall.

3 - The format of the "DateTime" field in the Meteotemplate database is "yyyy-mm-dd hh:mm:ss" and in the Excel/Netatmo files it is "yyyy/mm/dd hh:mm:ss".
Without much effort, I can change it in my CSV files. But is it necessary or convenient?

4 - Finally, the most important thing, can I export the data from CSV to SQL in two processes? The first one to export the unified temperature, humidity, pressure and wind file. The second one, for the rain file.
If two export processes are possible, I don't need to manipulate my rain CSV file.

I apologize for the length of this message and my limitations with English.
Thank you very much and Merry Christmas!

Buon Natale Davide.

[davidefa]

1) no need to delete the time stamp column ( in the import process any unwanted column is simply skipped )
2) no problem if the files are of different length, for the rain rate I can add an option to calculate the rain rate from the rain data ( this could be useful for others too )
3) the format of the dataTime input filter is not a problem as it is completely configurable, and will be automatically converted to the database format
4) as said in point 2) this should not be a problem

[El_Viejo]

Thank you very much for your help.

So far I have done these operations:
1 - In the temperature, humidity, pressure and wind sensor data, I have modified the original files by changing the reading interval to 10 minutes.
2 - In the original rain (R) file, I have calculated and added the "RR" data and, as far as possible, reduced the original reading intervals.

But, due to my limitations with English and with the export process, I still have a doubt.
The data obtained from the netatmo servers are in 4 different CSV files.
Is it possible to perform four different export processes, one for each CSV file, or is it necessary to group these four files into a single file and then perform a single export process?

Best regards.

[davidefa]

Yes, you can import ( you wrote export but I think you meant import ) the 4 files separately ( no need to merge in a single file ).
Please post/pm me your csv files, I will check if everything is correct ( sometimes a few lines contains some unexpected chars that block the import phase ) and I will show you how to configure the importer correctly for your data

[El_Viejo]

Thanks to davidefa's detailed instructions I have been able to import the first four Netatmo files, to update the SQL database of my Meteotemplate.

I will continue with the data for the month of August until I manage to import the data for the 100 days during which it was not possible for Meteotemplate to connect to the Netatmo servers.

Thank you very much Davide for your patience.