App Memory / Memory Leak FreePBX 2.10 BUG

Hello All,
I'm running the latest of everything from the distro and then upgraded via the shell script upgrade-1.88.210.57-1.sh.
I verify that everything is updated both via yum and module admin, but over the course of 24 hours or so you can see via the web interface the App memory increasing into the red, eventually causing voice quality issues.
I spent a couple of hours looking for anything that made sense with no luck, until I finally decided to reboot the apache process being at that time the biggest user of memory and that was it, memory dropped from 93% to 14% ( 2GB Server ).
I can duplicate this on multiple servers and several times in a row so the question becomes is it something FreePBX is doing or is it apache.
I can see all the servers are running:
httpd-2.2.19-1.w5
httpd-tools-2.2.19-1.w5
I tried to force a reinstall in case maybe something got corrupted, but all I get is Installed package httpd-2.2.19-1.w5.i386 not available.
Any help appreciated.



I am experiencing the same
I am experiencing the same issue on my servers. To temp fix the problem I setup a cron job to restart httpd...it's a ugly to do it, but its a temp fix to the problem.
Are you running anything
Are you running anything like FOP, FOP2 or other 3rd party applications. This should not be occurring and I have not seen this on any of our systems nor on any support customers as of yet.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
I'm not running anything
I'm not running anything extra or 3rd party...
I was debating setting up the cron job idea but was hoping to find something to fix the problem before trying that.
I have several servers and not all of them are experiencing this.
I have a case where I have 2 identical servers for different clients and one is having the problem to the point that it will within 24 hours crash and the other is running perfect. They are both upgraded from 2.9x.
My hunch is that there is some configuration entry which for lack of a better term is corrupted, has additional or wrong info which may be causing this.
One other note is on one of them both the primary and backup server have the same issue which is part of whats leaning me towards thinking a potential configuration type of issue.
Updates
I just saw there are 5 updates, I will run those and see if by chance things get better unless anyone has some suggestions. Updates just released:
Backup & Restore 2.10.0.14 will be upgraded to online version 2.10.0.15
Core 2.10.0rc1.0 will be upgraded to online version 2.10.0rc1.1
FreePBX Framework 2.10.0rc1.0 will be upgraded to online version 2.10.0rc1.1
Paging and Intercom 2.10.0.2 will be upgraded to online version 2.10.0.3
Queues 2.10.0.1 will be upgraded to online version 2.10.0.3
We have the same issue with App Memory Usage Increasing
We are experiencing the same issues. Our App Memory was at 11% (2GB Installed) in the morning and by evening had risen to 54%. By the next morning it was 72%. I shut down the system and installed another 2 GB of ram and it dropped to 4%. Now this morning it is up to 29% already. Not sure what to do now or what is causing it. We are running 2.10 FreePBX (all latest basic updates installed)
losbravos
At this point simply restart apache and instead of restarting the box and loosing / dropping calls until a fix comes out.
Tony, if you want I can give you access to a box where we can duplicate this all day long.
Please get a print out of
Please get a print out of top so we can see the processes and what each process is using for ram.
Also can you verify the php version and apache version are the same between the server with the issue and the server without the issue. Also are both of your systems built from the same Distro and kernel version?
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
Info
Hi Tony,
Both boxes are identical, apache version httpd-2.2.19-1.w5 and php php-5.3.3-1.3
The were all built from the same 2.9x and upgrade the same exact way via scripts.
There are 2 PDF links below from one of the boxes with the problem using ps -auxw
Results with 83% memory usage
Results after restart apache with 12% memory usage
It's Asterisk. Something is
It's Asterisk. Something is up. My first instinct would be a reinstall. Let's see what other suggestions are offered.
Hi SkykingOH, I don't
Hi SkykingOH,
I don't believe asterisk is the issue, it's inconsistent across servers both physical and virtual but is duplicated so it seems difficult to believe there were several bad installs.
I tried a reinstall of just asterisk, no difference I also restarted the asterisk process while the memory was at 93% and no change, of course thats not a completely accurate way to test that.
More memory info
93% Memory {{{ Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 2208 380 ? Ss Feb10 0:03 init [3] root 2 0.0 0.0 0 0 ? S Feb10 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S Feb10 0:00 [ksoftirqd/0] root 4 0.0 0.0 0 0 ? S Feb10 0:03 [migration/0] root 5 0.0 0.0 0 0 ? S Feb10 0:03 [migration/1] root 6 0.0 0.0 0 0 ? S Feb10 0:00 [ksoftirqd/1] root 7 0.0 0.0 0 0 ? S Feb10 0:09 [events/0] root 8 0.0 0.0 0 0 ? S Feb10 0:18 [events/1] root 9 0.0 0.0 0 0 ? S Feb10 0:05 [khelper] root 12 0.0 0.0 0 0 ? S Feb10 0:00 [netns] root 13 0.0 0.0 0 0 ? S Feb10 0:00 [async/mgr] root 16 0.0 0.0 0 0 ? S Feb10 0:00 [xenwatch] root 17 0.0 0.0 0 0 ? S Feb10 0:00 [xenbus] root 60 0.0 0.0 0 0 ? S Feb10 0:01 [sync_supers] root 62 0.0 0.0 0 0 ? S Feb10 0:00 [bdi-default] root 64 0.0 0.0 0 0 ? S Feb10 0:12 [kblockd/0] root 65 0.0 0.0 0 0 ? S Feb10 0:10 [kblockd/1] root 70 0.0 0.0 0 0 ? S Feb10 0:00 [kseriod] root 178 5.5 0.0 0 0 ? S Feb10 258:55 [kswapd0] root 179 0.0 0.0 0 0 ? S Feb10 0:00 [aio/0] root 180 0.0 0.0 0 0 ? S Feb10 0:00 [aio/1] root 181 0.0 0.0 0 0 ? S Feb10 0:00 [crypto/0] root 182 0.0 0.0 0 0 ? S Feb10 0:00 [crypto/1] root 193 0.0 0.0 0 0 ? S Feb10 0:00 [khvcd] root 219 0.0 0.0 0 0 ? S Feb10 0:17 [kjournald] root 243 0.0 0.0 0 0 ? S Feb10 0:00 [kauditd] root 271 0.0 0.0 0 0 ? S Feb10 0:03 [flush-202:0] root 277 0.0 0.0 2444 340 ? S<s Feb10 0:00 /sbin/udevd -d root 672 0.0 0.0 0 0 ? S Feb10 0:00 [kjournald] root 1072 0.0 0.0 10624 356 ? S<sl Feb10 0:01 auditd root 1074 0.0 0.0 11184 292 ? S<sl Feb10 0:01 /sbin/audispd root 1094 0.0 0.0 1860 408 ? Ss Feb10 0:12 syslogd -m 0 root 1099 0.0 0.0 1808 352 ? Ss Feb10 0:00 klogd -x root 1126 0.0 0.0 2608 312 ? Ss Feb10 0:17 irqbalance rpc 1162 0.0 0.0 1956 388 ? Ss Feb10 0:00 portmap rpcuser 1189 0.0 0.1 2012 576 ? Ss Feb10 0:00 rpc.statd dbus 1230 0.0 0.0 2896 364 ? Ss Feb10 0:00 dbus-daemon --system root 1292 0.0 0.0 10824 316 ? Ssl Feb10 0:14 pcscd 68 1366 0.0 0.1 5220 536 ? Ss Feb10 0:00 hald root 1367 0.0 0.0 3308 284 ? S Feb10 0:00 hald-runner root 1418 0.0 0.1 7256 640 ? Ss Feb10 0:25 /usr/sbin/sshd root 1439 0.0 0.0 2884 496 ? Ss Feb10 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid ntp 1450 0.0 0.8 4552 4548 ? SLs Feb10 0:15 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g root 1484 0.0 0.1 4680 572 ? S Feb10 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock - root 1601 0.0 0.1 7012 968 ? Ss Feb10 0:03 /usr/libexec/postfix/master postfix 1610 0.0 0.1 7232 732 ? S Feb10 0:00 qmgr -l -t fifo -u root 1611 0.0 0.0 2040 148 ? Ss Feb10 0:00 gpm -m /dev/input/mice -t exps2 root 1655 0.0 0.0 5384 500 ? Ss Feb10 0:01 crond xfs 1674 0.0 0.0 3440 372 ? Ss Feb10 0:00 xfs -droppriv -daemon root 1706 0.1 0.3 92916 1712 ? Sl Feb10 5:12 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock root 1708 0.0 0.1 2836 724 ? S Feb10 1:07 /usr/libexec/gam_server root 1768 0.0 0.0 2408 208 ? Ss Feb10 0:00 /usr/sbin/atd root 1787 0.0 0.0 3104 460 ? Ss Feb10 0:00 incrond avahi 1796 0.0 0.0 2736 376 ? Ss Feb10 0:00 avahi-daemon: running [bpbx.local] avahi 1797 0.0 0.0 2736 128 ? Ss Feb10 0:00 avahi-daemon: chroot helper nobody 1814 0.0 0.0 1984 324 ? S Feb10 0:00 dnsmasq asterisk 1920 0.0 0.0 4632 176 ? S Feb10 0:00 bash -c cd /var/www/html/admin/modules/fw_fop && /var/www/html/admin/modules/fw_fop/safe_o asterisk 1921 0.0 0.1 4628 520 ? S Feb10 0:00 sh /var/www/html/admin/modules/fw_fop/safe_opserver asterisk 1923 0.1 0.6 13052 3472 ? S Feb10 8:16 /usr/bin/perl /var/www/html/admin/modules/fw_fop/op_server.pl root 1935 0.0 0.0 3652 152 ? S Feb10 0:00 /usr/sbin/smartd -q never root 1938 0.0 0.0 1792 392 tty1 Ss+ Feb10 0:00 /sbin/mingetty tty1 root 2191 0.0 0.2 10084 1328 ? Ss Feb11 0:00 sshd: root@pts/0 root 2195 0.0 0.1 4676 540 pts/0 Ss+ Feb11 0:00 -bash asterisk 13253 1.6 4.7 47332 24420 ? S 06:21 0:59 /usr/sbin/httpd postfix 15568 0.0 0.2 7076 1124 ? S 06:46 0:00 pickup -l -t fifo -u asterisk 17568 2.2 2.4 35168 12512 ? S 07:18 0:02 /usr/sbin/httpd root 17716 0.2 0.5 10084 2860 ? Ss 07:19 0:00 sshd: root@pts/2 root 17722 2.3 0.2 4676 1340 pts/2 Ss 07:19 0:00 -bash root 17761 0.0 0.1 4396 952 pts/2 R+ 07:19 0:00 ps -auxw mysql 30421 0.2 0.6 123188 3468 ? Sl Feb11 4:44 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/r root 31284 0.0 0.0 4624 140 ? S Feb12 0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asterisk -g asterisk 31287 0.9 0.3 43284 1916 ? Sl Feb12 10:45 /usr/sbin/asterisk -f -U asterisk -G asterisk -g -vvvg -c root 31427 0.0 0.2 26104 1088 ? Ss Feb12 0:03 /usr/sbin/httpd asterisk 31431 1.5 11.3 149860 58316 ? S Feb12 17:31 /usr/sbin/httpd asterisk 31432 1.5 11.3 149916 58408 ? S Feb12 17:35 /usr/sbin/httpd asterisk 31433 1.5 11.2 148928 57824 ? S Feb12 17:33 /usr/sbin/httpd asterisk 31434 1.5 11.6 150812 60032 ? S Feb12 17:31 /usr/sbin/httpd asterisk 31435 1.5 11.8 153952 60920 ? S Feb12 17:35 /usr/sbin/httpd asterisk 31436 1.5 11.6 150068 59868 ? S Feb12 17:29 /usr/sbin/httpd asterisk 31485 1.5 11.9 153800 61632 ? S Feb12 17:28 /usr/sbin/httpd after rebooting apache 9% Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 2208 384 ? Ss Feb10 0:03 init [3] root 2 0.0 0.0 0 0 ? S Feb10 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S Feb10 0:00 [ksoftirqd/0] root 4 0.0 0.0 0 0 ? S Feb10 0:03 [migration/0] root 5 0.0 0.0 0 0 ? S Feb10 0:03 [migration/1] root 6 0.0 0.0 0 0 ? S Feb10 0:00 [ksoftirqd/1] root 7 0.0 0.0 0 0 ? S Feb10 0:09 [events/0] root 8 0.0 0.0 0 0 ? S Feb10 0:18 [events/1] root 9 0.0 0.0 0 0 ? S Feb10 0:05 [khelper] root 12 0.0 0.0 0 0 ? S Feb10 0:00 [netns] root 13 0.0 0.0 0 0 ? S Feb10 0:00 [async/mgr] root 16 0.0 0.0 0 0 ? S Feb10 0:00 [xenwatch] root 17 0.0 0.0 0 0 ? S Feb10 0:00 [xenbus] root 60 0.0 0.0 0 0 ? S Feb10 0:01 [sync_supers] root 62 0.0 0.0 0 0 ? S Feb10 0:00 [bdi-default] root 64 0.0 0.0 0 0 ? S Feb10 0:12 [kblockd/0] root 65 0.0 0.0 0 0 ? S Feb10 0:10 [kblockd/1] root 70 0.0 0.0 0 0 ? S Feb10 0:00 [kseriod] root 178 5.5 0.0 0 0 ? S Feb10 258:56 [kswapd0] root 179 0.0 0.0 0 0 ? S Feb10 0:00 [aio/0] root 180 0.0 0.0 0 0 ? S Feb10 0:00 [aio/1] root 181 0.0 0.0 0 0 ? S Feb10 0:00 [crypto/0] root 182 0.0 0.0 0 0 ? S Feb10 0:00 [crypto/1] root 193 0.0 0.0 0 0 ? S Feb10 0:00 [khvcd] root 219 0.0 0.0 0 0 ? S Feb10 0:17 [kjournald] root 243 0.0 0.0 0 0 ? S Feb10 0:00 [kauditd] root 271 0.0 0.0 0 0 ? S Feb10 0:03 [flush-202:0] root 277 0.0 0.0 2444 344 ? S<s Feb10 0:00 /sbin/udevd -d root 672 0.0 0.0 0 0 ? S Feb10 0:00 [kjournald] root 1072 0.0 0.0 10624 308 ? S<sl Feb10 0:01 auditd root 1074 0.0 0.0 11184 276 ? S<sl Feb10 0:01 /sbin/audispd root 1094 0.0 0.0 1860 392 ? Ss Feb10 0:12 syslogd -m 0 root 1099 0.0 0.0 1808 352 ? Ss Feb10 0:00 klogd -x root 1126 0.0 0.0 2608 320 ? Ss Feb10 0:17 irqbalance rpc 1162 0.0 0.0 1956 388 ? Ss Feb10 0:00 portmap rpcuser 1189 0.0 0.1 2012 580 ? Ss Feb10 0:00 rpc.statd dbus 1230 0.0 0.0 2896 368 ? Ss Feb10 0:00 dbus-daemon --system root 1292 0.0 0.0 10824 316 ? Ssl Feb10 0:14 pcscd 68 1366 0.0 0.1 5220 540 ? Ss Feb10 0:00 hald root 1367 0.0 0.0 3308 284 ? S Feb10 0:00 hald-runner root 1418 0.0 0.1 7256 640 ? Ss Feb10 0:25 /usr/sbin/sshd root 1439 0.0 0.0 2884 500 ? Ss Feb10 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid ntp 1450 0.0 0.8 4552 4548 ? SLs Feb10 0:15 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g root 1484 0.0 0.1 4680 576 ? S Feb10 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock - root 1601 0.0 0.1 7012 916 ? Ss Feb10 0:03 /usr/libexec/postfix/master postfix 1610 0.0 0.1 7232 732 ? S Feb10 0:00 qmgr -l -t fifo -u root 1611 0.0 0.0 2040 148 ? Ss Feb10 0:00 gpm -m /dev/input/mice -t exps2 root 1655 0.0 0.0 5384 476 ? Ss Feb10 0:01 crond xfs 1674 0.0 0.0 3440 372 ? Ss Feb10 0:00 xfs -droppriv -daemon root 1706 0.1 0.3 92916 1968 ? Sl Feb10 5:12 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock root 1708 0.0 0.1 2836 724 ? S Feb10 1:07 /usr/libexec/gam_server root 1768 0.0 0.0 2408 212 ? Ss Feb10 0:00 /usr/sbin/atd root 1787 0.0 0.0 3104 460 ? Ss Feb10 0:00 incrond avahi 1796 0.0 0.0 2736 376 ? Ss Feb10 0:00 avahi-daemon: running [bpbx.local] avahi 1797 0.0 0.0 2736 132 ? Ss Feb10 0:00 avahi-daemon: chroot helper nobody 1814 0.0 0.0 1984 328 ? S Feb10 0:00 dnsmasq asterisk 1920 0.0 0.0 4632 180 ? S Feb10 0:00 bash -c cd /var/www/html/admin/modules/fw_fop && /var/www/html/admin/modules/fw_fop/safe_o asterisk 1921 0.0 0.1 4628 524 ? S Feb10 0:00 sh /var/www/html/admin/modules/fw_fop/safe_opserver asterisk 1923 0.1 0.7 13052 3740 ? S Feb10 8:17 /usr/bin/perl /var/www/html/admin/modules/fw_fop/op_server.pl root 1935 0.0 0.0 3652 152 ? S Feb10 0:00 /usr/sbin/smartd -q never root 1938 0.0 0.0 1792 396 tty1 Ss+ Feb10 0:00 /sbin/mingetty tty1 root 2191 0.0 0.2 10084 1328 ? Ss Feb11 0:00 sshd: root@pts/0 root 2195 0.0 0.1 4676 540 pts/0 Ss+ Feb11 0:00 -bash postfix 15568 0.0 0.2 7076 1080 ? S 06:46 0:00 pickup -l -t fifo -u root 17716 0.0 0.5 10084 2572 ? Rs 07:19 0:00 sshd: root@pts/2 root 17722 0.1 0.2 4676 1420 pts/2 Ss 07:19 0:00 -bash root 17865 0.3 1.5 26104 7984 ? Ss 07:20 0:00 /usr/sbin/httpd asterisk 17867 3.9 3.2 37684 16524 ? S 07:20 0:00 /usr/sbin/httpd asterisk 17868 0.0 0.9 26104 4992 ? S 07:20 0:00 /usr/sbin/httpd asterisk 17869 1.1 1.9 31636 10216 ? S 07:20 0:00 /usr/sbin/httpd asterisk 17870 0.0 0.9 26104 4992 ? S 07:20 0:00 /usr/sbin/httpd asterisk 17871 0.0 0.9 26104 4992 ? S 07:20 0:00 /usr/sbin/httpd asterisk 17872 0.0 0.9 26104 4992 ? S 07:20 0:00 /usr/sbin/httpd asterisk 17873 0.0 0.9 26104 4992 ? S 07:20 0:00 /usr/sbin/httpd asterisk 17874 0.4 1.7 28916 9096 ? R 07:20 0:00 /usr/sbin/httpd root 17889 0.0 0.1 4396 948 pts/2 R+ 07:20 0:00 ps -auxw mysql 30421 0.2 0.6 123188 3472 ? Sl Feb11 4:45 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/r root 31284 0.0 0.0 4624 144 ? S Feb12 0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asterisk -g asterisk 31287 0.9 0.3 43284 1912 ? Sl Feb12 10:46 /usr/sbin/asterisk -f -U asterisk -G asterisk -g -vvvg -c }}}Please, when posting this
Please, when posting this type of output in the forums use the code tags. I have edited your post.
You posted three pages of unreadable text that trashed the formatting on the page.
For lurkers, click "input format" at the bottom of the message input box, then "bbcode help"
Thanks skyking, was flying
Thanks skyking, was flying out the door but wanted the extra info before restarting apache on that server, I put {{{ and }}} which is from another board for entering code, pressed post and ran out the door.
FYI both of those entries are from one server and the entries I posted the other day are from a different server.
I wonder if it is a php 5.3
I wonder if it is a php 5.3 issue. I have not had time to dig into this yet as I am not able to see the same issues on any of our systems. Does anyone else have this issue and if so can we get your php version. Also anyone running FreePBX 2.10 not on php 5.3 are you having this issue?
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
I have one of my backup
I have one of my backup systems I can give you access to and if you trash it, thats fine since it's a virtual box I can re provision it.
It's up to you.
I can also downgrade php assuming it wouldn't break anything, but I believe if your running 2.10 and you upgraded via scripts just about everyone will be on the same version of php.
Can you take a older distro
Can you take a older distro and just update FreePBX from the GUI and not use the upgrade scripts. That way it stays on php 5.2 and report back if you still have the problems.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
I have seen this problem
I have seen this problem once on a 2.10 php 5.3 system, I rebooted and have not seen the problem again and try as I might I cant reproduce it. It seems to me that it is being caused by one particular process that is not always in use as the symptoms are not consistently present.
upgrade
While I cant kill the production boxes, I can try it on a backup box.
If I use scripts up to the latest 2.9x will that keep php at 5.2x ?
Also stonet, if I reboot boxes ( which I have done several times ) no effect.
I can duplicate this problem on multiple boxes all day long.
Yes go up to anything that
Yes go up to anything that ends in .55 in the version. Than upgrade to FreePBX 2.10 in the GUI and let it run.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
Manual Upgrade didnt go well
Hi Tony,
I built a box from scratch, upgraded to upgrade-1.87.29.55-2.sh.
Once that was done, I ran the manual update and got it to update just about everything until I finally got this error:
rror!close
Error: Did not receive valid response from server
XHR response code: 200 XHR responseText: undefined jQuery status:
After getting passed that, I ended up with another module update that wanted php 5.3 and wouldnt update.
After that I rebooted, was able to apply settings and then did a restore at which point the web ui crashed:
HTTP Error 500 (Internal Server Error): An unexpected condition was encountered while the server was attempting to fulfill the request.
I will give it one more attempt from scratch, I dont believe it will make a difference but want to make sure I didnt miss a step.
Second attempt
------------------------------------
This didnt go well either, not having good success running web interface when you cant upgrade to php 5.3 and you end up with a cripled web interface... steps below:
upgrade-1.8.1.4-2.sh
upgrade-1.8.1.4-3.sh
upgrade-1.8.2.0-1.sh
upgrade-1.8.2.0-2.sh
upgrade-1.86.29.55-1.sh
upgrade-1.87.29.55-1.sh
upgrade-1.87.29.55-2.sh
System Admin 2.9.0.19 Schmoozecom.com Online upgrade available (2.10.0.20)
Error:
System Admin cannot be upgraded:
PHP version 5.3.0 or higher is required, you have 5.2.17
PHP Component Zend Guard Loader is required but missing from you PHP installation.
Please try again after the dependencies have been installed.
--------------------------------------------------------------------------------
Reload failed because retrieve_conf encountered an error: 255
click here for more info
1 error(s) occurred, you should view the notification log on the dashboard or main screen to check for more details.
--------------------------------------------------------------------------------
Get passed this, and then interface simply wont work reboot or otherwise.
Upgrading the final script upgrade-1.88.210.57-1.sh gets everything working again, of course that bumps up to php 5.3 so that screws up the test of trying to run on the older version of php.
Tony, any ideas??
Tony, any ideas??
Sorry I have a bunch of
Sorry I have a bunch of things going on with my normal day job and have not had time to look into this yet. Hopefully next week
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
I have now seen this for my
I have now seen this for my own eyes. It appears to be a php 5.3 memory leak from what I can debug. Its not something specific to 2.10 but php 5.3 which was upgraded on the Distro at the same time. I am working on compiling a new php 5.3 with the latest and see if that solves the issue. If it does we will compile new RPM's and get an upgrade script out for it.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
It would be great if we
It would be great if we could go back to php 5.2 as astercc does not work with 5.3. just a suggestion!
No 5.2 is EOL and not
No 5.2 is EOL and not getting any type of updates and 5.4 is already out the door. Plus when we go to Centos 6.x soon it will require php 5.3 or newer. Sorry
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
Is the fix going to be available as a Module update?
I have been following the thread, as we have the same issues (I posted earlier). Since doubling our RAM (to 4GB) we are holding steady at 40% or so...from less than 10% when we first come in the morning, but still on the high side..
Ok well we should have some
Ok well we should have some test RPM's shortly. Would you be willing to install them and report you findings.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
yes, i have a test system I
yes, i have a test system I can test it on.
Can you get me SSH into the
Can you get me SSH into the box. Send me a PM and I will get php upgraded and you can report back your finding.
My 2.9 box that I upgraded just to 2.10 without upgrading PHP has not climbed at all in memory so it 100% either a php or apache package issue. I just finished packages the php 5.3.10-1.3 which is the latest 5.3 from php direct.
I have not installed them on a machine yet so this will be a perfect test.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
In the yum repos for
In the yum repos for 1.88.210.57 I pushed new php rpm's. You can try the following. I only did this for the 32bit for now. If this solves the problem I will get 64bit ones done and roll a upgrade script.
yum install php *
Than make sure you
chown -R asterisk:asterisk /var/lib/php/session
lastly restart the apache service
service httpd restart
Please report back your findings ASAP so we know if we need to keep digging or not.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
I get the following
I get the following error
Error Downloading Packages:
php-process-5.3.10-1.3.i386: failure: RPMS/php/5.3.10-1.3/php-process-5.3.10-1.3.i386.rpm from pbx: [Errno 256] No more mirrors to try.
php-pdo-5.3.10-1.3.i386: failure: RPMS/php/5.3.10-1.3/php-pdo-5.3.10-1.3.i386.rpm from pbx: [Errno 256] No more mirrors to try.
OK process looks good update wise.
I was able to upgrade without problem:
Of course memory went down on the restart of apache, so lets see how it goes over the next few hours.
Thanks !!!
Error FYI
I also received some errors on a second box ( inhouse live server ) but I think the mirrors havent updated fully:
yum install php *
Loaded plugins: fastestmirror, kmod
Determining fastest mirrors
addons | 951 B 00:00
http://yum.freepbxdistro.org/centos/5.7/addons/i386/repodata/primary.xml... [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
addons/primary | 202 B 00:00
http://yum.freepbxdistro.org/centos/5.7/os/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: base. Please verify its path and try again
Initial update NOT looking good
running for literally 15 minutes tops, I can see the box which we updated PHP on went from 14% memory to 31% and that box is doing nothing ( backup box ).
Working on my live box to see additional test results...
OK the yum servers had a
OK the yum servers had a issue syncing and they are re-syncing now.
As far as your results that not promising. Not sure where else to look as I do not see memory issues on 2.10 with older PHP.
What version of httpd are you using and what about the older 2.9 boxes you have where you have no memory issue. What version of httpd are they on.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
I dont have any 2.9x boxes with memory issues
Only 2.10x.
Memory still creeping up on that box with the last php update. Up 20 41% again, this box has ZERO activity it's a live backup box.
No I am asking take your 2.9
No I am asking take your 2.9 box with no memory issue and see what version of httpd you are running and than do the same for a 2.10 box that has the memory issue. I want to make sure they have the same apache version so we can rule that out.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
show me a print out of ps
show me a print out of ps aux | grep httpd
I show on my box no memory use to speak of after upgrading to the latest php an hour ago.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
Also are you running FOP or
Also are you running FOP or Aastra XML for phones at all. Both of those items would hit apache.
Tony Lewis
Schmooze Com, Inc.
FreePBX Developer
Not running either of those
Whatever FOP is in the default install is in there.
The same server we upgraded PHP on yesterday is at this moment at 93% memory.
Do you want me to get you access, or should I reboot the process?
It's a backup system not live so we can do a few things if necessary.
Update
For those of you that are experiencing this same problem, can you please answer the following questions:
1 - Are running a "physical" server or a VPS?
2 - If it's a real server can you please tell me what brand and model it is?
3 - How much memory do you have?
4 - Do you still have the problem?
Thank you!
Confirm Bug in 2.10
VoIPTek, same issue with 1.89.210.57-2.
After one day of operation (without telephone load but with a browser remained open on the management window) memory is near to 100% and swap continuously increasing.
Did you find any solution?
Thanks
Carlo
top - 23:17:35 up 1 day, 4:51, 1 user, load average: 0.42, 0.40, 0.36
Tasks: 105 total, 2 running, 103 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 384860k total, 373104k used, 11756k free, 5116k buffers
Swap: 779144k total, 349024k used, 430120k free, 38344k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 15 0 2160 556 536 S 0.0 0.1 0:00.38 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.52 events/0
6 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
7 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
10 root 10 -5 0 0 0 S 0.0 0.0 0:00.97 kblockd/0
11 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
48 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0
51 root 10 -5 0 0 0 S 0.0 0.0 0:00.05 khubd
53 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
117 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
118 root 15 0 0 0 0 S 0.0 0.0 0:06.54 pdflush
119 root 15 0 0 0 0 S 0.0 0.0 0:09.28 pdflush
120 root 10 -5 0 0 0 S 0.0 0.0 0:00.96 kswapd0
121 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
270 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
290 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 ata/0
291 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 ata_aux
294 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
301 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kstriped
310 root 10 -5 0 0 0 S 0.0 0.0 0:22.67 kjournald
335 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kauditd
368 root 21 -4 2396 360 360 S 0.0 0.1 0:00.52 udevd
772 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 iprt/0
1131 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/0
1132 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kmpath_handlerd
1189 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kjournald
1321 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 iscsi_eh
1349 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cnic_wq
1369 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ib_addr
1376 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ib_mcast
1377 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ib_inform
1378 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 local_sa
1381 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 iw_cm_wq
1386 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ib_cm/0
1388 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 rdma_cm
1404 root 24 0 32704 280 280 S 0.0 0.1 0:00.00 brcm_iscsiuio
1410 root 15 0 3704 340 320 S 0.0 0.1 0:04.75 iscsid
1411 root 5 -10 4160 4156 3156 S 0.0 1.1 0:00.03 iscsid
1755 root 15 -4 12628 616 532 S 0.0 0.2 0:00.07 auditd
1757 root 7 -8 12164 628 576 S 0.0 0.2 0:00.02 audispd
1787 root 16 0 1816 520 476 S 0.0 0.1 0:01.12 syslogd
1790 root 15 0 1764 320 320 S 0.0 0.1 0:00.00 klogd
1875 rpc 21 0 1912 424 424 S 0.0 0.1 0:00.00 portmap
1910 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/0
1916 rpcuser 24 0 1964 592 592 S 0.0 0.2 0:00.00 rpc.statd
1952 root 25 0 5936 240 240 S 0.0 0.1 0:00.00 rpc.idmapd
1975 dbus 15 0 2848 864 724 S 0.0 0.2 0:00.09 dbus-daemon
2007 root 22 0 2256 552 552 S 0.0 0.1 0:00.00 hcid
2018 root 23 0 1832 368 368 S 0.0 0.1 0:00.00 sdpd
2031 root 9 -10 0 0 0 S 0.0 0.0 0:00.00 krfcommd
2075 root 25 0 12828 520 428 S 0.0 0.1 0:03.70 pcscd
2089 root 18 0 1760 448 448 S 0.0 0.1 0:00.00 acpid
2206 haldaemo 15 0 5984 1628 1308 S 0.0 0.4 0:30.82 hald
2207 root 25 0 3264 912 912 S 0.0 0.2 0:00.03 hald-runner
2216 haldaemo 25 0 2108 692 692 S 0.0 0.2 0:00.00 hald-addon-acpi
2222 haldaemo 25 0 2108 688 688 S 0.0 0.2 0:00.00 hald-addon-keyb
2235 root 16 0 2060 608 560 S 0.0 0.2 2:11.55 hald-addon-stor
2256 root 23 0 2008 336 336 S 0.0 0.1 0:00.00 hidd
2295 root 21 0 29424 940 832 S 0.0 0.2 0:01.50 automount
2382 root 25 0 8660 552 472 S 0.0 0.1 0:06.64 VBoxService
2405 root 15 0 7212 756 660 S 0.0 0.2 0:01.93 sshd
2421 root 18 0 2836 556 556 S 0.0 0.1 0:00.01 xinetd
2437 ntp 15 0 4512 4508 3496 S 0.0 1.2 0:00.26 ntpd
2474 root 25 0 4632 1028 1028 S 0.0 0.3 0:00.02 mysqld_safe
2524 mysql 15 0 137m 4688 2624 S 0.0 1.2 1:02.77 mysqld
2596 root 20 0 6968 1492 1420 S 0.0 0.4 0:00.55 master
2606 postfix 15 0 7092 1468 1404 S 0.0 0.4 0:00.30 qmgr
2611 root 18 0 2000 236 236 S 0.0 0.1 0:00.00 gpm
2625 root 15 0 25824 4340 3572 S 0.0 1.1 0:01.42 httpd
2654 root 25 0 4632 396 396 S 0.0 0.1 0:00.00 safe_asterisk
2666 asterisk 15 0 49644 5020 3580 S 0.0 1.3 1:31.91 asterisk
2672 root 18 0 5380 624 556 S 0.0 0.2 0:00.19 crond
2689 asterisk 18 0 94392 37m 3536 S 0.0 9.8 2:29.32 httpd
2690 asterisk 15 0 94908 35m 3832 S 0.0 9.6 2:29.79 httpd
2691 asterisk 20 0 94236 36m 3564 S 0.0 9.7 2:26.74 httpd
2692 asterisk 22 0 94908 36m 3548 S 0.0 9.7 2:29.30 httpd
2693 asterisk 18 0 94072 36m 3548 S 0.0 9.7 2:29.08 httpd
2694 asterisk 18 0 94452 37m 3836 S 0.0 9.9 2:27.76 httpd
2695 asterisk 21 0 95228 38m 3536 S 0.0 10.1 2:27.96 httpd
2696 asterisk 21 0 94332 37m 3544 S 0.0 9.9 2:28.77 httpd
2704 xfs 18 0 3400 576 544 S 0.0 0.1 0:00.00 xfs
2730 root 18 0 110m 2808 1580 S 0.0 0.7 0:25.48 fail2ban-server
2732 root 15 0 2768 1052 896 S 0.0 0.3 0:12.77 gam_server
2801 root 24 0 4628 1008 1008 S 0.0 0.3 0:00.01 startup.sh
2822 root 24 0 4632 1020 1020 S 0.0 0.3 0:00.01 bash
2838 root 18 0 2360 352 312 S 0.0 0.1 0:00.00 atd
2841 root 19 0 255m 5052 1268 S 0.0 1.3 0:10.71 java
2863 root 24 0 3068 508 508 S 0.0 0.1 0:00.00 incrond
2890 nobody 18 0 1916 316 292 S 0.0 0.1 0:00.02 dnsmasq
2995 root 34 19 25904 3576 2116 S 0.0 0.9 0:00.33 yum-updatesd
3039 root 15 0 3612 252 236 S 0.0 0.1 0:00.00 smartd
3042 root 17 0 1748 396 396 S 0.0 0.1 0:00.01 mingetty
3043 root 18 0 1748 376 376 S 0.0 0.1 0:00.00 mingetty
3044 root 18 0 1748 396 396 S 0.0 0.1 0:00.00 mingetty
3045 root 19 0 1748 376 376 S 0.0 0.1 0:00.00 mingetty
3046 root 20 0 1748 396 396 S 0.0 0.1 0:00.00 mingetty
3047 root 21 0 1748 376 376 S 0.0 0.1 0:00.00 mingetty
7043 postfix 18 0 7032 1816 1464 S 0.0 0.5 0:00.07 pickup
9716 asterisk 18 0 41876 19m 3672 S 0.0 5.3 0:04.88 httpd
12517 root 15 0 2408 996 784 R 0.0 0.3 0:00.01 top
16424 root 15 0 10200 2452 2228 R 0.0 0.6 0:00.48 sshd
16431 root 15 0 4636 1308 1148 S 0.0 0.3 0:00.09 bash
Another bit
The bug is clearly related to the use of the System Administration GUI.
In a night, keeping any Management window carefully closed, the system didn't increase memory usage.
What "System Administration
What "System Administration GUI" display page did you have opened that you observed this?
If you can create a 'reasonably reproducible" situation please feel free to file a bug so one of the developers can have a closer look.
Please make sure to provide details of FreePBX, apache, php, etc. versions that are being run in the ticket.
Philippe Lindheimer - FreePBX Project Leader
FreePBX Training Opportunities - Click Here
Get Official Paid Support - Click Here
Status Page
The browser was open on the FreePBX auto-reloading status page.
So - when left unadvertently open- it reloads at least one hundred times in a night.
This is the reason why IMHO it is a reasonable suspect, because the page auto-generates http sessions.
Thanks
Carlo
not sure...
One of my initial hunches was that keeping the status report open was possibly using up the memory, however on boxes that I was looking at frequently when I left the management page in the evening and opened it the next day memory DID increase buy a good jump.
With all the recent updates I have seen an improvement, memory is not disappearing as quickly. I will leave my servers status screen open today and see if we can confirm that one way or another.
crogialli, I see your current and I think at the moment that's about all you can do. I would also say stay out of the status or for that matter the entire GUI to see if that improves the situation.
Are you running a real server or are you running a VPS?
Running in a VPS
It's in a Virtual Machine, over a Mac OSX server.
Until it's fixed, I'll try to stay out the gui and I think I'll schedule a machine restart every night at 3:00 am.
dont need to restart
you don't need to restart the whole system, simply restart apache and the memory clears up.
I'll follow your suggestion
Thanks VoIPTek, sure I will follow your advice.
Carlo
Philippe
When this was first opened memory was disappearing at an alarming rate to the point that you would have to reboot apache daily. After all the updates from when this was originally opened, I'm not sure the problem still exists.
The screen I believe that people run is the FreePBX System status which also happens to tell you the percentage of memory.
In my testing today keeping the GUI status open I do see I went from 20-22% but thats nothing, I would literally run out of memory within a day, so that has improved with all the updates. Now it may not be related to the GUI because I did have the problems as stated previously.
I don't believe the problem is there at the level it was before because I could duplicate it on 6 different servers and right now I really cant say I can duplicate it.
crogialli, run yum check-update from the command line of the server, does it tell you that you need any updates? I know you ran the scripts already and should be current, this is just a way to double check. Also after your updates have you rebooted?