Too much data in the DB ?

Post Reply
linux150
Observer
Observer
Posts: 15
Joined: Sat May 22, 2021 12:18 pm
Station model: WMR968
Software: SDR -> Weewx -> MT
Contact:

Too much data in the DB ?

Post by linux150 » Fri Jun 18, 2021 9:48 pm

Hi!
I found a way to export my history data from Meteohub via WD-Export. Then i cumulated the rain with an formular in Excel and then i import all data. My problem now is, that my site is very slow.
On the serverside i did contact the helpdesk about the slow response, and he switched me to a faster one.
Now i think my database is the problem, because the export of WD create an entry for every minute!
So my datas are from 2009 and now i have about 6.000.000 !! Data entrys in my DB.
And so i think the slow response from my site is because the big sql DB ?!
And why do i post this in programming? Well i want to ask if theres any way in SQL to get less DB Entrys in calculating a average of, for example, 5 Minutes.
Means for me, select DB Entry from 00:00 to 00:05 (5 entrys) calculate an average and write on entry (maybe in another table).
I hope you know what i mean. I have the RAW WD CSV Files (for each month) also but not with the cumulated rain. This was also a lot of work, so i ask for a solution in SQL. Maybe calculating once more in Excel would be easier ...?

Sorry for my english, i hope u understand my problem ....

If you want test my site you find in my profil.

LG
Martin

User avatar
John B
Forecaster
Forecaster
Posts: 211
Joined: Sun Aug 20, 2017 11:59 pm
Location: Blaxland, N.S.W., Australia
Station model: La Crosse WS2355
Software: Weather Display
Contact:

Re: Too much data in the DB ?

Post by John B » Sat Jun 19, 2021 2:26 am

Hello, Martin,

This might help:
http://www.meteotemplate.com/web/wiki/wiki4GB.php

On the left side select Data Import (if it is not already). You might find what you are seeking there.

Good luck,
John

linux150
Observer
Observer
Posts: 15
Joined: Sat May 22, 2021 12:18 pm
Station model: WMR968
Software: SDR -> Weewx -> MT
Contact:

Re: Too much data in the DB ?

Post by linux150 » Sat Jun 19, 2021 8:08 pm

Thanks, exactly what i need :-)
I did read most of the wiki but i think i was blind a little bit.
I will try and hope it will boost my site a little bit, because in the moment it is lame.

LG
Martin

linux150
Observer
Observer
Posts: 15
Joined: Sat May 22, 2021 12:18 pm
Station model: WMR968
Software: SDR -> Weewx -> MT
Contact:

Re: Too much data in the DB ?

Post by linux150 » Sun Jun 20, 2021 2:07 pm

Automatic didn’t work, I did get an error from my host. Something about my .htaccess
But didn’t matter, I backup the alldata and renamed new to alldata.
All work fine now
Tnx agn
Martin

User avatar
John B
Forecaster
Forecaster
Posts: 211
Joined: Sun Aug 20, 2017 11:59 pm
Location: Blaxland, N.S.W., Australia
Station model: La Crosse WS2355
Software: Weather Display
Contact:

Re: Too much data in the DB ?

Post by John B » Sun Jun 20, 2021 7:12 pm

Yes, Martin, it seems a bit complicated, even though it is not. I found that reading through that part of the Wiki until I understood which parts needed were important. I was fortunate as it worked first time. :)

As I said last time, good luck.

John

PNCATTE
Observer
Observer
Posts: 22
Joined: Wed Oct 10, 2018 8:05 am
Location: FLEURIE
Station model: Davis Pro 2
Software: WLIP
Contact:

Re: Too much data in the DB ?

Post by PNCATTE » Wed Aug 04, 2021 2:15 pm

Hello,

Following a problem with my host I lost the history of my database
As I had saved a data file in .CSV I wanted to reload it from the control panel mode in Data Import but it gives me an error code:
E 406 The browser does not accept the MIME type of the requested page.

do you have any idea how to load the data

Thanks in advance
erreur E  406.png
erreur E 406.png (140.35 KiB) Viewed 2134 times
Attachments
Data Import.png
Data Import.png (287.34 KiB) Viewed 2134 times

davidefa
Expert
Expert
Posts: 652
Joined: Tue Jan 12, 2021 8:03 am
Location: Italy
Station model: WH2650
Software: WH2650 (direct upload)
Contact:

Re: Too much data in the DB ?

Post by davidefa » Thu Aug 05, 2021 2:12 pm

I've tried your setup and works for me ( with firefox )

e406.png
e406.png (84.8 KiB) Viewed 2116 times
( both with and without leading 'www.' )


I've searched about error 406 and found that even if the error is 'client side' generated the source may be more difficult to pinpoint ( should be some 'incomprehension' between server and browser ).
I suggest the following:
- try using a different browser; even starting the browser in 'clean mode' ( without extension enabled ) may be worth trying
- try using a different computer/operating system
( if you don't have access to a different computer you may download a virtualizing software such as vmware, a virtual machine with linux already installed and repeat the import from within the virtual ambient )
Image

PNCATTE
Observer
Observer
Posts: 22
Joined: Wed Oct 10, 2018 8:05 am
Location: FLEURIE
Station model: Davis Pro 2
Software: WLIP
Contact:

Re: Too much data in the DB ?

Post by PNCATTE » Thu Aug 05, 2021 5:25 pm

Hello,

Thanks for this information, I tried with three other browsers and a MAC ditto
I can't find my problem if I launch the link:
http://fleurie-meteolive.fr/MTE/install/alldata2013.csv

I see downloading the data file

Can you try by accessing my site with the password: 12345 to see if it is not from a corrupted file on my server?

http://fleurie-meteolive.fr/MTE/indexDesktop.php


Thank you for your help
Pascal

davidefa
Expert
Expert
Posts: 652
Joined: Tue Jan 12, 2021 8:03 am
Location: Italy
Station model: WH2650
Software: WH2650 (direct upload)
Contact:

Re: Too much data in the DB ?

Post by davidefa » Thu Aug 05, 2021 7:03 pm

The 12345 password does not work for me
Image

PNCATTE
Observer
Observer
Posts: 22
Joined: Wed Oct 10, 2018 8:05 am
Location: FLEURIE
Station model: Davis Pro 2
Software: WLIP
Contact:

Re: Too much data in the DB ?

Post by PNCATTE » Fri Aug 06, 2021 6:32 am

Hello,

For the URL you just have to note:
../install/alldata2013.csv

Thank you again for the information

Kind regards

johnathan
Observer
Observer
Posts: 22
Joined: Thu Apr 08, 2021 1:14 pm
Station model: wh1080
Software: WU

Re: Too much data in the DB ?

Post by johnathan » Tue Aug 24, 2021 11:23 am

The app development process involves creating an app from scratch. The original idea for the idea may be to create a social networking application, but app development company https://mlsdev.com/blog/53-how-much-doe ... mobile-app can help you with a variety of different ideas. Some experts can help you design a game or a tool for social interaction, while others can help you build shopping carts or virtual walls for a real brick-and-mortar shop. Others can help you create the back end of your site or the front end, like web applications and email functionality. They can also help you design the interface and develop the look and feel of your web pages.

Post Reply