Webcam pictures used too much disk space

[Luc]

My Internet Service Provider told me I had exceeded my allowed disk space (of 2 GB) with 600 %.
90 % of the total space on my web server was occupied by the webcam images and thumb nails.
For testing purposes I have several instances of meteotemplate running and they all used their own space for the web images.
I decided to remove the webcam plugin from meteotemplate for the above mentioned reasons.

[Jachym]

Hi Luc,
sorry to hear that, unfortunately there is obviously nothing we can do in terms of the plugin. Only thing I can think of is reducing the quality of the images that you send to the server. Though this of course means degrading the image quality. High quality = large size, this unfortunately will always hold (at least until some AI comes up with some genius solution )

[nitrx]

I delete all images at the end of the month not very genius, unfortunally dropbox hasn't a facility anymore I think.

[Jachym]

Deleting the images from the folder is not ideal because this only deletes the actual image, but not the reference to it in the database, so visitors will still see these days as if images were available. You need to remove the reference from the mySQL table for the webcam as well.

[Luc]

I deleted the webcam plugin too!

[nitrx]

Well indeed I forget to say I drop the table and delete the thumbnails too. I've a copy on my SSD

[Jachym]

I understand the reasons, but I am not plannnig any changes. This is probably the only way it can work. If you want to archive images you need webspace. Some hosts offer unlimited, so at least for some people this might work.

[Thomas]

Hello,
I take this post to bring my experience with the plugin webcams.
I have 3 webcams that work with this plugin (30 minutes frequency)
I do not have disk space problems but file management.
If I want to open the images folder in the plugin with my FTP client this is impossible.
If I want to open the file by http it's very slow.
Today I do not know how to manage all these files.
Would it be possible to store them per month for example ?

[Jachym]

Theoretically yes, but I should have thought about this when creating this plugin. I dont thing it would be possible to now change it because PHP would run out of time when trying to do it if you have too many files.
The other issue with this plugin is that I cannot test it myself, I remember it was very difficult to do and I decided I will not be creating plugins and blocks which I cannot test myself anymore. Which unfortunately is also the reason why there are no plugins and not much stuff for the extra sensors - but at least you have them now in the db and you can use this data however you want it later.

[Zedekia]

Hi Folks,

my hoster have a file-limitation. (mx. 264000 files). My upload-frequency is 30 m (17520 images per yr).

This works for me....I have no problems with performance ....

cu.

Zedekia

[Jachym]

If I was doing it today I would have split it into folders, the problem is that I dont think PHP can handle moving so many files now that people have had this plugin for many months.

[WessexWeather]

This is a great plugin.
Initially I was running the cron job every 5 minutes but I soon realised this would quickly consume my disk space, so I now save every 15 minutes. This saves space and still produces a good time lapse in the plugin and block.
I do agree that sorting images into folders would be a good idea, and would be happy to start over with this plugin if this was implemented in a new version.

[Jachym]

Implementing this would require basically starting from scratch with the images - it would not use what is already available. I mean it would work if you manually re-organized the files, but that seems too time-consuming and complicated to do for some Im sure. So this is a sort of dead end situation.

When deciding about the interval you should obviously also keep in mind the actual resolution of the image. Saving 4K images is different than saving VGA ones.

[EveshamWeather]

I love the webcam plugin and luckily my current webhost provides 'unlimited' storage space so that is not an issue for me. I am suffering from file overload in the single folder though as my host (and all the previous hosts it seems - possibly all Cpanel hosts) seems to have a hard limit of 9998 files visible in each folder using FTP. The other files are there, you just cannot see any above that point.

I may look into the webcam plugin and make a fork of it to improve the file handling (use folders) if I get the time and Jachym doesn't mind. Unlike Jachym, who needs to think about how a change affects everyone, I am lucky and it would only affect me, so manual sorting of previous images will be possible. Will take me a while though as I am not a PHP programmer so need to learn that first .

If anyone else looks at updating the plugin, these are my 'wish list' items

• Improved file handling by using year/month/day folder structure

• A way to remove some historic images automatically - for instance after 3 months remove all but 1 file per hour. This will help with space for those with more limited webhosting

[WessexWeather]

Hi Vince,

I agree with you 100%. It is so tedious to find a recent image if you need it, given the file display limitations - I cannot display on my iPhone or iPad any files later than 30th July 2017 which is a real pain! My workaround is to find the image in the plugin, long press to open the image in a new tab and save it to Camera Roll from there.

A far better solution would be to store the images in date-stamped folders. Jachym says this is possible but would mean a rewrite of the plugin. For me I would happily delete the whole lot and start again for this functionality.

[Jachym]

The major issue with this plugin (and with several other plugins and blocks) is that I cannot really use it myself. The DEMO uses some static examples, but just like in some other blocks, I can for example never test any "current images" where my image is already outdated. Likewise, my folder has 3 images.... so trying some script to sort the data can never be tested properly.
There were times when I said to myself I will never again develop anything that I cannot test. But few days later you get another email and give it a try. If I actually did that there would be nothing related to NetAtmo, WANET, WeatherLink, any extra sensor including UV, solar, no webcam stuff, no Bloomsky, no WxSim....
Its great when it works and in most cases it does, however when there is a problem it ususally really "is a problem" given I cannot test the solution thoroughly enough.
Not that this would be your problem, I totally realize this is my problem, but you happen to be unfortunate in that you use something I do

[Jachym]

This sorting script... I have an idea how to do it, but would it really work? And I cannot afford to release a solution that I have no idea how it will really behave. You can test on a few images, but thats just not enough.

[Luc]

May be we should introduce the concept of alpha and beta testing done by a small group of enthousiastic testers who don't mind things won't work in one go.
Then when all is tested thoroughly we (I mean you, Jachym) can make it public.

[Jachym]

thats how it more or less works, but it is actually in some cases complicated to come up with the initial script at all, most problematic are scripts showing current data (because even if I get some files from someone they are soon not current) and with data that require the db (extra sensors) because you cant just use random generator for that and my provider doesnt allow access to external dbs (in case I wanted to use someone's data).
We will see. I will look at this when I have time, Im relatively busy with other things right now, but I can put it on my to-do list

[Luc]

Hi Jachym,
I could send data of my weather station including extra data like soil moisture and -temp to the api of your test-environment (with its own test database). Also I could upload with ftp web photos in a frequency you would prefer (you need to create a ftp account for me then and I need to know your test upload password).
Luc

[Jachym]

FTP is probably not an option, there are many restrictions on my provider's side, including country etc.

API would probably be possible, my station is useless these days anyway... it was not measuring this morning and now is offline again I think. Everytime the contacts get wet, it stops sending data

[EveshamWeather]

Jachym, if I can help with testing, images etc just let me know when you get time to look at this. Will do what I can to help.

[Jachym]

Two things have to be done in this case:
1. script that will sort the pictures into directories and then empty the root one
2. change paths in all the scripts using these images

[WessexWeather]

I would also be happy to test new scripts for this plugin. If it messes things up, no worries, I'd be happy to delete all my images and start over. Let me know.

[Jachym]

OK, actually, when you first mentioned it here I remembered there was some problem...now I remember.

The issue is that the way this script would work is:
1. get list of available images
2. create new folder for each month/day
3. move the images to the corresponding folders
4. delete the original ones

The problem is in step 1.... as you know, many people, including me, use external webhosting. If you have been using that plugin for quite some time it is possible that PHP will not be able to load the files list into memory and the whole thing will crash before even starting. And the memory cannot usually be increased when you are not using your own server.