Too much data in the DB ?
-
- Observer
- Posts: 23
- Joined: Sat May 22, 2021 12:18 pm
- Station model: WMR968
- Software: SDR -> Weewx -> MT
- Contact:
Too much data in the DB ?
Hi!
I found a way to export my history data from Meteohub via WD-Export. Then i cumulated the rain with an formular in Excel and then i import all data. My problem now is, that my site is very slow.
On the serverside i did contact the helpdesk about the slow response, and he switched me to a faster one.
Now i think my database is the problem, because the export of WD create an entry for every minute!
So my datas are from 2009 and now i have about 6.000.000 !! Data entrys in my DB.
And so i think the slow response from my site is because the big sql DB ?!
And why do i post this in programming? Well i want to ask if theres any way in SQL to get less DB Entrys in calculating a average of, for example, 5 Minutes.
Means for me, select DB Entry from 00:00 to 00:05 (5 entrys) calculate an average and write on entry (maybe in another table).
I hope you know what i mean. I have the RAW WD CSV Files (for each month) also but not with the cumulated rain. This was also a lot of work, so i ask for a solution in SQL. Maybe calculating once more in Excel would be easier ...?
Sorry for my english, i hope u understand my problem ....
If you want test my site you find in my profil.
LG
Martin
I found a way to export my history data from Meteohub via WD-Export. Then i cumulated the rain with an formular in Excel and then i import all data. My problem now is, that my site is very slow.
On the serverside i did contact the helpdesk about the slow response, and he switched me to a faster one.
Now i think my database is the problem, because the export of WD create an entry for every minute!
So my datas are from 2009 and now i have about 6.000.000 !! Data entrys in my DB.
And so i think the slow response from my site is because the big sql DB ?!
And why do i post this in programming? Well i want to ask if theres any way in SQL to get less DB Entrys in calculating a average of, for example, 5 Minutes.
Means for me, select DB Entry from 00:00 to 00:05 (5 entrys) calculate an average and write on entry (maybe in another table).
I hope you know what i mean. I have the RAW WD CSV Files (for each month) also but not with the cumulated rain. This was also a lot of work, so i ask for a solution in SQL. Maybe calculating once more in Excel would be easier ...?
Sorry for my english, i hope u understand my problem ....
If you want test my site you find in my profil.
LG
Martin
- John B
- Forecaster
- Posts: 211
- Joined: Sun Aug 20, 2017 11:59 pm
- Location: Blaxland, N.S.W., Australia
- Station model: La Crosse WS2355
- Software: Weather Display
- Contact:
Re: Too much data in the DB ?
Hello, Martin,
This might help:
http://www.meteotemplate.com/web/wiki/wiki4GB.php
On the left side select Data Import (if it is not already). You might find what you are seeking there.
Good luck,
John
This might help:
http://www.meteotemplate.com/web/wiki/wiki4GB.php
On the left side select Data Import (if it is not already). You might find what you are seeking there.
Good luck,
John
-
- Observer
- Posts: 23
- Joined: Sat May 22, 2021 12:18 pm
- Station model: WMR968
- Software: SDR -> Weewx -> MT
- Contact:
Re: Too much data in the DB ?
Thanks, exactly what i need
I did read most of the wiki but i think i was blind a little bit.
I will try and hope it will boost my site a little bit, because in the moment it is lame.
LG
Martin
I did read most of the wiki but i think i was blind a little bit.
I will try and hope it will boost my site a little bit, because in the moment it is lame.
LG
Martin
-
- Observer
- Posts: 23
- Joined: Sat May 22, 2021 12:18 pm
- Station model: WMR968
- Software: SDR -> Weewx -> MT
- Contact:
Re: Too much data in the DB ?
Automatic didn’t work, I did get an error from my host. Something about my .htaccess
But didn’t matter, I backup the alldata and renamed new to alldata.
All work fine now
Tnx agn
Martin
But didn’t matter, I backup the alldata and renamed new to alldata.
All work fine now
Tnx agn
Martin
- John B
- Forecaster
- Posts: 211
- Joined: Sun Aug 20, 2017 11:59 pm
- Location: Blaxland, N.S.W., Australia
- Station model: La Crosse WS2355
- Software: Weather Display
- Contact:
Re: Too much data in the DB ?
Yes, Martin, it seems a bit complicated, even though it is not. I found that reading through that part of the Wiki until I understood which parts needed were important. I was fortunate as it worked first time.
As I said last time, good luck.
John
As I said last time, good luck.
John
-
- Observer
- Posts: 22
- Joined: Wed Oct 10, 2018 8:05 am
- Location: FLEURIE
- Station model: Davis Pro 2
- Software: WLIP
- Contact:
Re: Too much data in the DB ?
Hello,
Following a problem with my host I lost the history of my database
As I had saved a data file in .CSV I wanted to reload it from the control panel mode in Data Import but it gives me an error code:
E 406 The browser does not accept the MIME type of the requested page.
do you have any idea how to load the data
Thanks in advance
Following a problem with my host I lost the history of my database
As I had saved a data file in .CSV I wanted to reload it from the control panel mode in Data Import but it gives me an error code:
E 406 The browser does not accept the MIME type of the requested page.
do you have any idea how to load the data
Thanks in advance
- Attachments
-
- Data Import.png (287.34 KiB) Viewed 4842 times
-
- Expert
- Posts: 862
- Joined: Tue Jan 12, 2021 8:03 am
- Location: Italy
- Station model: WH2650
- Software: WH2650 (direct upload)
- Contact:
Re: Too much data in the DB ?
I've tried your setup and works for me ( with firefox )
( both with and without leading 'www.' )
I've searched about error 406 and found that even if the error is 'client side' generated the source may be more difficult to pinpoint ( should be some 'incomprehension' between server and browser ).
I suggest the following:
- try using a different browser; even starting the browser in 'clean mode' ( without extension enabled ) may be worth trying
- try using a different computer/operating system
( if you don't have access to a different computer you may download a virtualizing software such as vmware, a virtual machine with linux already installed and repeat the import from within the virtual ambient )
( both with and without leading 'www.' )
I've searched about error 406 and found that even if the error is 'client side' generated the source may be more difficult to pinpoint ( should be some 'incomprehension' between server and browser ).
I suggest the following:
- try using a different browser; even starting the browser in 'clean mode' ( without extension enabled ) may be worth trying
- try using a different computer/operating system
( if you don't have access to a different computer you may download a virtualizing software such as vmware, a virtual machine with linux already installed and repeat the import from within the virtual ambient )
-
- Observer
- Posts: 22
- Joined: Wed Oct 10, 2018 8:05 am
- Location: FLEURIE
- Station model: Davis Pro 2
- Software: WLIP
- Contact:
Re: Too much data in the DB ?
Hello,
Thanks for this information, I tried with three other browsers and a MAC ditto
I can't find my problem if I launch the link:
http://fleurie-meteolive.fr/MTE/install/alldata2013.csv
I see downloading the data file
Can you try by accessing my site with the password: 12345 to see if it is not from a corrupted file on my server?
http://fleurie-meteolive.fr/MTE/indexDesktop.php
Thank you for your help
Pascal
Thanks for this information, I tried with three other browsers and a MAC ditto
I can't find my problem if I launch the link:
http://fleurie-meteolive.fr/MTE/install/alldata2013.csv
I see downloading the data file
Can you try by accessing my site with the password: 12345 to see if it is not from a corrupted file on my server?
http://fleurie-meteolive.fr/MTE/indexDesktop.php
Thank you for your help
Pascal
-
- Observer
- Posts: 22
- Joined: Wed Oct 10, 2018 8:05 am
- Location: FLEURIE
- Station model: Davis Pro 2
- Software: WLIP
- Contact:
Re: Too much data in the DB ?
Hello,
For the URL you just have to note:
../install/alldata2013.csv
Thank you again for the information
Kind regards
For the URL you just have to note:
../install/alldata2013.csv
Thank you again for the information
Kind regards
Re: Too much data in the DB ?
The app development process involves creating an app from scratch. The original idea for the idea may be to create a social networking application, but app development company https://mlsdev.com/blog/53-how-much-doe ... mobile-app can help you with a variety of different ideas. Some experts can help you design a game or a tool for social interaction, while others can help you build shopping carts or virtual walls for a real brick-and-mortar shop. Others can help you create the back end of your site or the front end, like web applications and email functionality. They can also help you design the interface and develop the look and feel of your web pages.