App Memory / Memory Leak FreePBX 2.10 BUG

VoIPTek's picture

Hello All,

I'm running the latest of everything from the distro and then upgraded via the shell script upgrade-1.88.210.57-1.sh.
I verify that everything is updated both via yum and module admin, but over the course of 24 hours or so you can see via the web interface the App memory increasing into the red, eventually causing voice quality issues.

I spent a couple of hours looking for anything that made sense with no luck, until I finally decided to reboot the apache process being at that time the biggest user of memory and that was it, memory dropped from 93% to 14% ( 2GB Server ).

I can duplicate this on multiple servers and several times in a row so the question becomes is it something FreePBX is doing or is it apache.

I can see all the servers are running:
httpd-2.2.19-1.w5
httpd-tools-2.2.19-1.w5

I tried to force a reinstall in case maybe something got corrupted, but all I get is Installed package httpd-2.2.19-1.w5.i386 not available.

Any help appreciated.


__________________


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

I am experiencing the same

wvroger's picture

I am experiencing the same issue on my servers. To temp fix the problem I setup a cron job to restart httpd...it's a ugly to do it, but its a temp fix to the problem.


Are you running anything

tonyclewis's picture

Are you running anything like FOP, FOP2 or other 3rd party applications. This should not be occurring and I have not seen this on any of our systems nor on any support customers as of yet.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


I'm not running anything

VoIPTek's picture

I'm not running anything extra or 3rd party...
I was debating setting up the cron job idea but was hoping to find something to fix the problem before trying that.

I have several servers and not all of them are experiencing this.
I have a case where I have 2 identical servers for different clients and one is having the problem to the point that it will within 24 hours crash and the other is running perfect. They are both upgraded from 2.9x.

My hunch is that there is some configuration entry which for lack of a better term is corrupted, has additional or wrong info which may be causing this.

One other note is on one of them both the primary and backup server have the same issue which is part of whats leaning me towards thinking a potential configuration type of issue.


Updates

VoIPTek's picture

I just saw there are 5 updates, I will run those and see if by chance things get better unless anyone has some suggestions. Updates just released:

Backup & Restore 2.10.0.14 will be upgraded to online version 2.10.0.15
Core 2.10.0rc1.0 will be upgraded to online version 2.10.0rc1.1
FreePBX Framework 2.10.0rc1.0 will be upgraded to online version 2.10.0rc1.1
Paging and Intercom 2.10.0.2 will be upgraded to online version 2.10.0.3
Queues 2.10.0.1 will be upgraded to online version 2.10.0.3


We have the same issue with App Memory Usage Increasing

losbravos's picture

We are experiencing the same issues. Our App Memory was at 11% (2GB Installed) in the morning and by evening had risen to 54%. By the next morning it was 72%. I shut down the system and installed another 2 GB of ram and it dropped to 4%. Now this morning it is up to 29% already. Not sure what to do now or what is causing it. We are running 2.10 FreePBX (all latest basic updates installed)


losbravos

VoIPTek's picture

At this point simply restart apache and instead of restarting the box and loosing / dropping calls until a fix comes out.

Tony, if you want I can give you access to a box where we can duplicate this all day long.


Please get a print out of

tonyclewis's picture

Please get a print out of top so we can see the processes and what each process is using for ram.

Also can you verify the php version and apache version are the same between the server with the issue and the server without the issue. Also are both of your systems built from the same Distro and kernel version?


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


Info

VoIPTek's picture

Hi Tony,

Both boxes are identical, apache version httpd-2.2.19-1.w5 and php php-5.3.3-1.3

The were all built from the same 2.9x and upgrade the same exact way via scripts.

There are 2 PDF links below from one of the boxes with the problem using ps -auxw

Results with 83% memory usage

Results after restart apache with 12% memory usage


It's Asterisk. Something is

SkykingOH's picture

It's Asterisk. Something is up. My first instinct would be a reinstall. Let's see what other suggestions are offered.


Hi SkykingOH, I don't

VoIPTek's picture

Hi SkykingOH,

I don't believe asterisk is the issue, it's inconsistent across servers both physical and virtual but is duplicated so it seems difficult to believe there were several bad installs.

I tried a reinstall of just asterisk, no difference I also restarted the asterisk process while the memory was at 93% and no change, of course thats not a completely accurate way to test that.


More memory info

VoIPTek's picture
93% Memory
{{{
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2208   380 ?        Ss   Feb10   0:03 init [3]  
root         2  0.0  0.0      0     0 ?        S    Feb10   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Feb10   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    Feb10   0:03 [migration/0]
root         5  0.0  0.0      0     0 ?        S    Feb10   0:03 [migration/1]
root         6  0.0  0.0      0     0 ?        S    Feb10   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S    Feb10   0:09 [events/0]
root         8  0.0  0.0      0     0 ?        S    Feb10   0:18 [events/1]
root         9  0.0  0.0      0     0 ?        S    Feb10   0:05 [khelper]
root        12  0.0  0.0      0     0 ?        S    Feb10   0:00 [netns]
root        13  0.0  0.0      0     0 ?        S    Feb10   0:00 [async/mgr]
root        16  0.0  0.0      0     0 ?        S    Feb10   0:00 [xenwatch]
root        17  0.0  0.0      0     0 ?        S    Feb10   0:00 [xenbus]
root        60  0.0  0.0      0     0 ?        S    Feb10   0:01 [sync_supers]
root        62  0.0  0.0      0     0 ?        S    Feb10   0:00 [bdi-default]
root        64  0.0  0.0      0     0 ?        S    Feb10   0:12 [kblockd/0]
root        65  0.0  0.0      0     0 ?        S    Feb10   0:10 [kblockd/1]
root        70  0.0  0.0      0     0 ?        S    Feb10   0:00 [kseriod]
root       178  5.5  0.0      0     0 ?        S    Feb10 258:55 [kswapd0]
root       179  0.0  0.0      0     0 ?        S    Feb10   0:00 [aio/0]
root       180  0.0  0.0      0     0 ?        S    Feb10   0:00 [aio/1]
root       181  0.0  0.0      0     0 ?        S    Feb10   0:00 [crypto/0]
root       182  0.0  0.0      0     0 ?        S    Feb10   0:00 [crypto/1]
root       193  0.0  0.0      0     0 ?        S    Feb10   0:00 [khvcd]
root       219  0.0  0.0      0     0 ?        S    Feb10   0:17 [kjournald]
root       243  0.0  0.0      0     0 ?        S    Feb10   0:00 [kauditd]
root       271  0.0  0.0      0     0 ?        S    Feb10   0:03 [flush-202:0]
root       277  0.0  0.0   2444   340 ?        S<s  Feb10   0:00 /sbin/udevd -d
root       672  0.0  0.0      0     0 ?        S    Feb10   0:00 [kjournald]
root      1072  0.0  0.0  10624   356 ?        S<sl Feb10   0:01 auditd
root      1074  0.0  0.0  11184   292 ?        S<sl Feb10   0:01 /sbin/audispd
root      1094  0.0  0.0   1860   408 ?        Ss   Feb10   0:12 syslogd -m 0
root      1099  0.0  0.0   1808   352 ?        Ss   Feb10   0:00 klogd -x
root      1126  0.0  0.0   2608   312 ?        Ss   Feb10   0:17 irqbalance
rpc       1162  0.0  0.0   1956   388 ?        Ss   Feb10   0:00 portmap
rpcuser   1189  0.0  0.1   2012   576 ?        Ss   Feb10   0:00 rpc.statd
dbus      1230  0.0  0.0   2896   364 ?        Ss   Feb10   0:00 dbus-daemon --system
root      1292  0.0  0.0  10824   316 ?        Ssl  Feb10   0:14 pcscd
68        1366  0.0  0.1   5220   536 ?        Ss   Feb10   0:00 hald
root      1367  0.0  0.0   3308   284 ?        S    Feb10   0:00 hald-runner
root      1418  0.0  0.1   7256   640 ?        Ss   Feb10   0:25 /usr/sbin/sshd
root      1439  0.0  0.0   2884   496 ?        Ss   Feb10   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp       1450  0.0  0.8   4552  4548 ?        SLs  Feb10   0:15 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      1484  0.0  0.1   4680   572 ?        S    Feb10   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock -
root      1601  0.0  0.1   7012   968 ?        Ss   Feb10   0:03 /usr/libexec/postfix/master
postfix   1610  0.0  0.1   7232   732 ?        S    Feb10   0:00 qmgr -l -t fifo -u
root      1611  0.0  0.0   2040   148 ?        Ss   Feb10   0:00 gpm -m /dev/input/mice -t exps2
root      1655  0.0  0.0   5384   500 ?        Ss   Feb10   0:01 crond
xfs       1674  0.0  0.0   3440   372 ?        Ss   Feb10   0:00 xfs -droppriv -daemon
root      1706  0.1  0.3  92916  1712 ?        Sl   Feb10   5:12 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
root      1708  0.0  0.1   2836   724 ?        S    Feb10   1:07 /usr/libexec/gam_server
root      1768  0.0  0.0   2408   208 ?        Ss   Feb10   0:00 /usr/sbin/atd
root      1787  0.0  0.0   3104   460 ?        Ss   Feb10   0:00 incrond
avahi     1796  0.0  0.0   2736   376 ?        Ss   Feb10   0:00 avahi-daemon: running [bpbx.local]
avahi     1797  0.0  0.0   2736   128 ?        Ss   Feb10   0:00 avahi-daemon: chroot helper
nobody    1814  0.0  0.0   1984   324 ?        S    Feb10   0:00 dnsmasq
asterisk  1920  0.0  0.0   4632   176 ?        S    Feb10   0:00 bash -c cd /var/www/html/admin/modules/fw_fop && /var/www/html/admin/modules/fw_fop/safe_o
asterisk  1921  0.0  0.1   4628   520 ?        S    Feb10   0:00 sh /var/www/html/admin/modules/fw_fop/safe_opserver
asterisk  1923  0.1  0.6  13052  3472 ?        S    Feb10   8:16 /usr/bin/perl /var/www/html/admin/modules/fw_fop/op_server.pl
root      1935  0.0  0.0   3652   152 ?        S    Feb10   0:00 /usr/sbin/smartd -q never
root      1938  0.0  0.0   1792   392 tty1     Ss+  Feb10   0:00 /sbin/mingetty tty1
root      2191  0.0  0.2  10084  1328 ?        Ss   Feb11   0:00 sshd: root@pts/0 
root      2195  0.0  0.1   4676   540 pts/0    Ss+  Feb11   0:00 -bash
asterisk 13253  1.6  4.7  47332 24420 ?        S    06:21   0:59 /usr/sbin/httpd
postfix  15568  0.0  0.2   7076  1124 ?        S    06:46   0:00 pickup -l -t fifo -u
asterisk 17568  2.2  2.4  35168 12512 ?        S    07:18   0:02 /usr/sbin/httpd
root     17716  0.2  0.5  10084  2860 ?        Ss   07:19   0:00 sshd: root@pts/2 
root     17722  2.3  0.2   4676  1340 pts/2    Ss   07:19   0:00 -bash
root     17761  0.0  0.1   4396   952 pts/2    R+   07:19   0:00 ps -auxw
mysql    30421  0.2  0.6 123188  3468 ?        Sl   Feb11   4:44 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/r
root     31284  0.0  0.0   4624   140 ?        S    Feb12   0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asterisk -g
asterisk 31287  0.9  0.3  43284  1916 ?        Sl   Feb12  10:45 /usr/sbin/asterisk -f -U asterisk -G asterisk -g -vvvg -c
root     31427  0.0  0.2  26104  1088 ?        Ss   Feb12   0:03 /usr/sbin/httpd
asterisk 31431  1.5 11.3 149860 58316 ?        S    Feb12  17:31 /usr/sbin/httpd
asterisk 31432  1.5 11.3 149916 58408 ?        S    Feb12  17:35 /usr/sbin/httpd
asterisk 31433  1.5 11.2 148928 57824 ?        S    Feb12  17:33 /usr/sbin/httpd
asterisk 31434  1.5 11.6 150812 60032 ?        S    Feb12  17:31 /usr/sbin/httpd
asterisk 31435  1.5 11.8 153952 60920 ?        S    Feb12  17:35 /usr/sbin/httpd
asterisk 31436  1.5 11.6 150068 59868 ?        S    Feb12  17:29 /usr/sbin/httpd
asterisk 31485  1.5 11.9 153800 61632 ?        S    Feb12  17:28 /usr/sbin/httpd


after rebooting apache 9%
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2208   384 ?        Ss   Feb10   0:03 init [3]  
root         2  0.0  0.0      0     0 ?        S    Feb10   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Feb10   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    Feb10   0:03 [migration/0]
root         5  0.0  0.0      0     0 ?        S    Feb10   0:03 [migration/1]
root         6  0.0  0.0      0     0 ?        S    Feb10   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S    Feb10   0:09 [events/0]
root         8  0.0  0.0      0     0 ?        S    Feb10   0:18 [events/1]
root         9  0.0  0.0      0     0 ?        S    Feb10   0:05 [khelper]
root        12  0.0  0.0      0     0 ?        S    Feb10   0:00 [netns]
root        13  0.0  0.0      0     0 ?        S    Feb10   0:00 [async/mgr]
root        16  0.0  0.0      0     0 ?        S    Feb10   0:00 [xenwatch]
root        17  0.0  0.0      0     0 ?        S    Feb10   0:00 [xenbus]
root        60  0.0  0.0      0     0 ?        S    Feb10   0:01 [sync_supers]
root        62  0.0  0.0      0     0 ?        S    Feb10   0:00 [bdi-default]
root        64  0.0  0.0      0     0 ?        S    Feb10   0:12 [kblockd/0]
root        65  0.0  0.0      0     0 ?        S    Feb10   0:10 [kblockd/1]
root        70  0.0  0.0      0     0 ?        S    Feb10   0:00 [kseriod]
root       178  5.5  0.0      0     0 ?        S    Feb10 258:56 [kswapd0]
root       179  0.0  0.0      0     0 ?        S    Feb10   0:00 [aio/0]
root       180  0.0  0.0      0     0 ?        S    Feb10   0:00 [aio/1]
root       181  0.0  0.0      0     0 ?        S    Feb10   0:00 [crypto/0]
root       182  0.0  0.0      0     0 ?        S    Feb10   0:00 [crypto/1]
root       193  0.0  0.0      0     0 ?        S    Feb10   0:00 [khvcd]
root       219  0.0  0.0      0     0 ?        S    Feb10   0:17 [kjournald]
root       243  0.0  0.0      0     0 ?        S    Feb10   0:00 [kauditd]
root       271  0.0  0.0      0     0 ?        S    Feb10   0:03 [flush-202:0]
root       277  0.0  0.0   2444   344 ?        S<s  Feb10   0:00 /sbin/udevd -d
root       672  0.0  0.0      0     0 ?        S    Feb10   0:00 [kjournald]
root      1072  0.0  0.0  10624   308 ?        S<sl Feb10   0:01 auditd
root      1074  0.0  0.0  11184   276 ?        S<sl Feb10   0:01 /sbin/audispd
root      1094  0.0  0.0   1860   392 ?        Ss   Feb10   0:12 syslogd -m 0
root      1099  0.0  0.0   1808   352 ?        Ss   Feb10   0:00 klogd -x
root      1126  0.0  0.0   2608   320 ?        Ss   Feb10   0:17 irqbalance
rpc       1162  0.0  0.0   1956   388 ?        Ss   Feb10   0:00 portmap
rpcuser   1189  0.0  0.1   2012   580 ?        Ss   Feb10   0:00 rpc.statd
dbus      1230  0.0  0.0   2896   368 ?        Ss   Feb10   0:00 dbus-daemon --system
root      1292  0.0  0.0  10824   316 ?        Ssl  Feb10   0:14 pcscd
68        1366  0.0  0.1   5220   540 ?        Ss   Feb10   0:00 hald
root      1367  0.0  0.0   3308   284 ?        S    Feb10   0:00 hald-runner
root      1418  0.0  0.1   7256   640 ?        Ss   Feb10   0:25 /usr/sbin/sshd
root      1439  0.0  0.0   2884   500 ?        Ss   Feb10   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp       1450  0.0  0.8   4552  4548 ?        SLs  Feb10   0:15 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      1484  0.0  0.1   4680   576 ?        S    Feb10   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock -
root      1601  0.0  0.1   7012   916 ?        Ss   Feb10   0:03 /usr/libexec/postfix/master
postfix   1610  0.0  0.1   7232   732 ?        S    Feb10   0:00 qmgr -l -t fifo -u
root      1611  0.0  0.0   2040   148 ?        Ss   Feb10   0:00 gpm -m /dev/input/mice -t exps2
root      1655  0.0  0.0   5384   476 ?        Ss   Feb10   0:01 crond
xfs       1674  0.0  0.0   3440   372 ?        Ss   Feb10   0:00 xfs -droppriv -daemon
root      1706  0.1  0.3  92916  1968 ?        Sl   Feb10   5:12 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
root      1708  0.0  0.1   2836   724 ?        S    Feb10   1:07 /usr/libexec/gam_server
root      1768  0.0  0.0   2408   212 ?        Ss   Feb10   0:00 /usr/sbin/atd
root      1787  0.0  0.0   3104   460 ?        Ss   Feb10   0:00 incrond
avahi     1796  0.0  0.0   2736   376 ?        Ss   Feb10   0:00 avahi-daemon: running [bpbx.local]
avahi     1797  0.0  0.0   2736   132 ?        Ss   Feb10   0:00 avahi-daemon: chroot helper
nobody    1814  0.0  0.0   1984   328 ?        S    Feb10   0:00 dnsmasq
asterisk  1920  0.0  0.0   4632   180 ?        S    Feb10   0:00 bash -c cd /var/www/html/admin/modules/fw_fop && /var/www/html/admin/modules/fw_fop/safe_o
asterisk  1921  0.0  0.1   4628   524 ?        S    Feb10   0:00 sh /var/www/html/admin/modules/fw_fop/safe_opserver
asterisk  1923  0.1  0.7  13052  3740 ?        S    Feb10   8:17 /usr/bin/perl /var/www/html/admin/modules/fw_fop/op_server.pl
root      1935  0.0  0.0   3652   152 ?        S    Feb10   0:00 /usr/sbin/smartd -q never
root      1938  0.0  0.0   1792   396 tty1     Ss+  Feb10   0:00 /sbin/mingetty tty1
root      2191  0.0  0.2  10084  1328 ?        Ss   Feb11   0:00 sshd: root@pts/0 
root      2195  0.0  0.1   4676   540 pts/0    Ss+  Feb11   0:00 -bash
postfix  15568  0.0  0.2   7076  1080 ?        S    06:46   0:00 pickup -l -t fifo -u
root     17716  0.0  0.5  10084  2572 ?        Rs   07:19   0:00 sshd: root@pts/2 
root     17722  0.1  0.2   4676  1420 pts/2    Ss   07:19   0:00 -bash
root     17865  0.3  1.5  26104  7984 ?        Ss   07:20   0:00 /usr/sbin/httpd
asterisk 17867  3.9  3.2  37684 16524 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17868  0.0  0.9  26104  4992 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17869  1.1  1.9  31636 10216 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17870  0.0  0.9  26104  4992 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17871  0.0  0.9  26104  4992 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17872  0.0  0.9  26104  4992 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17873  0.0  0.9  26104  4992 ?        S    07:20   0:00 /usr/sbin/httpd
asterisk 17874  0.4  1.7  28916  9096 ?        R    07:20   0:00 /usr/sbin/httpd
root     17889  0.0  0.1   4396   948 pts/2    R+   07:20   0:00 ps -auxw
mysql    30421  0.2  0.6 123188  3472 ?        Sl   Feb11   4:45 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/r
root     31284  0.0  0.0   4624   144 ?        S    Feb12   0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asterisk -g
asterisk 31287  0.9  0.3  43284  1912 ?        Sl   Feb12  10:46 /usr/sbin/asterisk -f -U asterisk -G asterisk -g -vvvg -c




}}}

Please, when posting this

SkykingOH's picture

Please, when posting this type of output in the forums use the code tags. I have edited your post.

You posted three pages of unreadable text that trashed the formatting on the page.

For lurkers, click "input format" at the bottom of the message input box, then "bbcode help"


Thanks skyking, was flying

VoIPTek's picture

Thanks skyking, was flying out the door but wanted the extra info before restarting apache on that server, I put {{{ and }}} which is from another board for entering code, pressed post and ran out the door.

FYI both of those entries are from one server and the entries I posted the other day are from a different server.


I wonder if it is a php 5.3

tonyclewis's picture

I wonder if it is a php 5.3 issue. I have not had time to dig into this yet as I am not able to see the same issues on any of our systems. Does anyone else have this issue and if so can we get your php version. Also anyone running FreePBX 2.10 not on php 5.3 are you having this issue?


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


I have one of my backup

VoIPTek's picture

I have one of my backup systems I can give you access to and if you trash it, thats fine since it's a virtual box I can re provision it.
It's up to you.
I can also downgrade php assuming it wouldn't break anything, but I believe if your running 2.10 and you upgraded via scripts just about everyone will be on the same version of php.


Can you take a older distro

tonyclewis's picture

Can you take a older distro and just update FreePBX from the GUI and not use the upgrade scripts. That way it stays on php 5.2 and report back if you still have the problems.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


I have seen this problem

stonet's picture

I have seen this problem once on a 2.10 php 5.3 system, I rebooted and have not seen the problem again and try as I might I cant reproduce it. It seems to me that it is being caused by one particular process that is not always in use as the symptoms are not consistently present.


upgrade

VoIPTek's picture

While I cant kill the production boxes, I can try it on a backup box.
If I use scripts up to the latest 2.9x will that keep php at 5.2x ?

Also stonet, if I reboot boxes ( which I have done several times ) no effect.

I can duplicate this problem on multiple boxes all day long.


Yes go up to anything that

tonyclewis's picture

Yes go up to anything that ends in .55 in the version. Than upgrade to FreePBX 2.10 in the GUI and let it run.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


Manual Upgrade didnt go well

VoIPTek's picture

Hi Tony,

I built a box from scratch, upgraded to upgrade-1.87.29.55-2.sh.
Once that was done, I ran the manual update and got it to update just about everything until I finally got this error:
rror!close

Error: Did not receive valid response from server

XHR response code: 200 XHR responseText: undefined jQuery status:

After getting passed that, I ended up with another module update that wanted php 5.3 and wouldnt update.

After that I rebooted, was able to apply settings and then did a restore at which point the web ui crashed:

HTTP Error 500 (Internal Server Error): An unexpected condition was encountered while the server was attempting to fulfill the request.

I will give it one more attempt from scratch, I dont believe it will make a difference but want to make sure I didnt miss a step.

Second attempt
------------------------------------

This didnt go well either, not having good success running web interface when you cant upgrade to php 5.3 and you end up with a cripled web interface... steps below:

upgrade-1.8.1.4-2.sh
upgrade-1.8.1.4-3.sh
upgrade-1.8.2.0-1.sh
upgrade-1.8.2.0-2.sh
upgrade-1.86.29.55-1.sh
upgrade-1.87.29.55-1.sh
upgrade-1.87.29.55-2.sh

System Admin 2.9.0.19 Schmoozecom.com Online upgrade available (2.10.0.20)
Error:
System Admin cannot be upgraded:

PHP version 5.3.0 or higher is required, you have 5.2.17
PHP Component Zend Guard Loader is required but missing from you PHP installation.

Please try again after the dependencies have been installed.

--------------------------------------------------------------------------------

Reload failed because retrieve_conf encountered an error: 255
click here for more info

1 error(s) occurred, you should view the notification log on the dashboard or main screen to check for more details.
--------------------------------------------------------------------------------

Get passed this, and then interface simply wont work reboot or otherwise.
Upgrading the final script upgrade-1.88.210.57-1.sh gets everything working again, of course that bumps up to php 5.3 so that screws up the test of trying to run on the older version of php.


Tony, any ideas??

VoIPTek's picture

Tony, any ideas??


Sorry I have a bunch of

tonyclewis's picture

Sorry I have a bunch of things going on with my normal day job and have not had time to look into this yet. Hopefully next week


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


I have now seen this for my

tonyclewis's picture

I have now seen this for my own eyes. It appears to be a php 5.3 memory leak from what I can debug. Its not something specific to 2.10 but php 5.3 which was upgraded on the Distro at the same time. I am working on compiling a new php 5.3 with the latest and see if that solves the issue. If it does we will compile new RPM's and get an upgrade script out for it.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


It would be great if we

wvroger's picture

It would be great if we could go back to php 5.2 as astercc does not work with 5.3. just a suggestion!


No 5.2 is EOL and not

tonyclewis's picture

No 5.2 is EOL and not getting any type of updates and 5.4 is already out the door. Plus when we go to Centos 6.x soon it will require php 5.3 or newer. Sorry


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


Is the fix going to be available as a Module update?

losbravos's picture

I have been following the thread, as we have the same issues (I posted earlier). Since doubling our RAM (to 4GB) we are holding steady at 40% or so...from less than 10% when we first come in the morning, but still on the high side..


Ok well we should have some

tonyclewis's picture

Ok well we should have some test RPM's shortly. Would you be willing to install them and report you findings.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


yes, i have a test system I

wvroger's picture

yes, i have a test system I can test it on.


Can you get me SSH into the

tonyclewis's picture

Can you get me SSH into the box. Send me a PM and I will get php upgraded and you can report back your finding.

My 2.9 box that I upgraded just to 2.10 without upgrading PHP has not climbed at all in memory so it 100% either a php or apache package issue. I just finished packages the php 5.3.10-1.3 which is the latest 5.3 from php direct.

I have not installed them on a machine yet so this will be a perfect test.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


In the yum repos for

tonyclewis's picture

In the yum repos for 1.88.210.57 I pushed new php rpm's. You can try the following. I only did this for the 32bit for now. If this solves the problem I will get 64bit ones done and roll a upgrade script.

yum install php *

Than make sure you

chown -R asterisk:asterisk /var/lib/php/session

lastly restart the apache service

service httpd restart

Please report back your findings ASAP so we know if we need to keep digging or not.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


I get the following

wvroger's picture

I get the following error

Error Downloading Packages:
php-process-5.3.10-1.3.i386: failure: RPMS/php/5.3.10-1.3/php-process-5.3.10-1.3.i386.rpm from pbx: [Errno 256] No more mirrors to try.
php-pdo-5.3.10-1.3.i386: failure: RPMS/php/5.3.10-1.3/php-pdo-5.3.10-1.3.i386.rpm from pbx: [Errno 256] No more mirrors to try.


OK process looks good update wise.

VoIPTek's picture

I was able to upgrade without problem:


===========================================================================================================================================================
 Package                                 Arch                              Version                                  Repository                        Size
===========================================================================================================================================================
Updating:
 php                                     i386                              5.3.10-1.3                               pbx                              1.3 M
Updating for dependencies:
 php-cli                                 i386                              5.3.10-1.3                               pbx                              2.5 M
 php-common                              i386                              5.3.10-1.3                               pbx                              626 k
 php-devel                               i386                              5.3.10-1.3                               pbx                              604 k
 php-gd                                  i386                              5.3.10-1.3                               pbx                              108 k
 php-mysql                               i386                              5.3.10-1.3                               pbx                               87 k
 php-pdo                                 i386                              5.3.10-1.3                               pbx                               66 k

Of course memory went down on the restart of apache, so lets see how it goes over the next few hours.

Thanks !!!


Error FYI

VoIPTek's picture

I also received some errors on a second box ( inhouse live server ) but I think the mirrors havent updated fully:
yum install php *
Loaded plugins: fastestmirror, kmod
Determining fastest mirrors
addons | 951 B 00:00
http://yum.freepbxdistro.org/centos/5.7/addons/i386/repodata/primary.xml... [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
addons/primary | 202 B 00:00
http://yum.freepbxdistro.org/centos/5.7/os/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: base. Please verify its path and try again


Initial update NOT looking good

VoIPTek's picture

running for literally 15 minutes tops, I can see the box which we updated PHP on went from 14% memory to 31% and that box is doing nothing ( backup box ).

Working on my live box to see additional test results...


OK the yum servers had a

tonyclewis's picture

OK the yum servers had a issue syncing and they are re-syncing now.

As far as your results that not promising. Not sure where else to look as I do not see memory issues on 2.10 with older PHP.

What version of httpd are you using and what about the older 2.9 boxes you have where you have no memory issue. What version of httpd are they on.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


I dont have any 2.9x boxes with memory issues

VoIPTek's picture

Only 2.10x.

Memory still creeping up on that box with the last php update. Up 20 41% again, this box has ZERO activity it's a live backup box.


No I am asking take your 2.9

tonyclewis's picture

No I am asking take your 2.9 box with no memory issue and see what version of httpd you are running and than do the same for a 2.10 box that has the memory issue. I want to make sure they have the same apache version so we can rule that out.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


show me a print out of ps

tonyclewis's picture

show me a print out of ps aux | grep httpd

I show on my box no memory use to speak of after upgrading to the latest php an hour ago.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


Also are you running FOP or

tonyclewis's picture

Also are you running FOP or Aastra XML for phones at all. Both of those items would hit apache.


__________________

Tony Lewis
Schmooze Com, Inc.
FreePBX Developer


Not running either of those

VoIPTek's picture

Whatever FOP is in the default install is in there.

The same server we upgraded PHP on yesterday is at this moment at 93% memory.
Do you want me to get you access, or should I reboot the process?
It's a backup system not live so we can do a few things if necessary.

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
root      3578  0.0  0.1  26156  1012 ?        Ss   Feb16   0:02 /usr/sbin/httpd
asterisk  3580  0.8  9.2 128828 47548 ?        S    Feb16   9:08 /usr/sbin/httpd
asterisk  3581  0.8  9.0 129112 46800 ?        S    Feb16   9:08 /usr/sbin/httpd
asterisk  3582  0.8  9.3 130092 48080 ?        S    Feb16   9:10 /usr/sbin/httpd
asterisk  3583  0.8  9.1 129068 46832 ?        S    Feb16   9:04 /usr/sbin/httpd
asterisk  3584  0.8  9.1 129064 47084 ?        S    Feb16   9:06 /usr/sbin/httpd
asterisk  3585  0.8  9.2 129284 47392 ?        S    Feb16   9:14 /usr/sbin/httpd
asterisk  3586  0.8  9.3 129356 47872 ?        S    Feb16   9:07 /usr/sbin/httpd
asterisk  3587  0.8  9.2 128524 47340 ?        S    Feb16   9:11 /usr/sbin/httpd
asterisk  6695  0.6  5.3  51532 27300 ?        S    07:58   1:05 /usr/sbin/httpd
root     20671  0.0  0.1   4056   684 pts/0    S+   10:39   0:00 grep -i httpd

Update

VoIPTek's picture

For those of you that are experiencing this same problem, can you please answer the following questions:

1 - Are running a "physical" server or a VPS?
2 - If it's a real server can you please tell me what brand and model it is?
3 - How much memory do you have?
4 - Do you still have the problem?

Thank you!


Confirm Bug in 2.10

crogialli's picture

VoIPTek, same issue with 1.89.210.57-2.
After one day of operation (without telephone load but with a browser remained open on the management window) memory is near to 100% and swap continuously increasing.

Did you find any solution?

Thanks

Carlo

top - 23:17:35 up 1 day, 4:51, 1 user, load average: 0.42, 0.40, 0.36
Tasks: 105 total, 2 running, 103 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 384860k total, 373104k used, 11756k free, 5116k buffers
Swap: 779144k total, 349024k used, 430120k free, 38344k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 15 0 2160 556 536 S 0.0 0.1 0:00.38 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.52 events/0
6 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
7 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
10 root 10 -5 0 0 0 S 0.0 0.0 0:00.97 kblockd/0
11 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
48 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0
51 root 10 -5 0 0 0 S 0.0 0.0 0:00.05 khubd
53 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
117 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
118 root 15 0 0 0 0 S 0.0 0.0 0:06.54 pdflush
119 root 15 0 0 0 0 S 0.0 0.0 0:09.28 pdflush
120 root 10 -5 0 0 0 S 0.0 0.0 0:00.96 kswapd0
121 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
270 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
290 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 ata/0
291 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 ata_aux
294 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
301 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kstriped
310 root 10 -5 0 0 0 S 0.0 0.0 0:22.67 kjournald
335 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kauditd
368 root 21 -4 2396 360 360 S 0.0 0.1 0:00.52 udevd
772 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 iprt/0
1131 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/0
1132 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kmpath_handlerd
1189 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kjournald
1321 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 iscsi_eh
1349 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cnic_wq
1369 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ib_addr
1376 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ib_mcast
1377 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ib_inform
1378 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 local_sa
1381 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 iw_cm_wq
1386 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ib_cm/0
1388 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 rdma_cm
1404 root 24 0 32704 280 280 S 0.0 0.1 0:00.00 brcm_iscsiuio
1410 root 15 0 3704 340 320 S 0.0 0.1 0:04.75 iscsid
1411 root 5 -10 4160 4156 3156 S 0.0 1.1 0:00.03 iscsid
1755 root 15 -4 12628 616 532 S 0.0 0.2 0:00.07 auditd
1757 root 7 -8 12164 628 576 S 0.0 0.2 0:00.02 audispd
1787 root 16 0 1816 520 476 S 0.0 0.1 0:01.12 syslogd
1790 root 15 0 1764 320 320 S 0.0 0.1 0:00.00 klogd
1875 rpc 21 0 1912 424 424 S 0.0 0.1 0:00.00 portmap
1910 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/0
1916 rpcuser 24 0 1964 592 592 S 0.0 0.2 0:00.00 rpc.statd
1952 root 25 0 5936 240 240 S 0.0 0.1 0:00.00 rpc.idmapd
1975 dbus 15 0 2848 864 724 S 0.0 0.2 0:00.09 dbus-daemon
2007 root 22 0 2256 552 552 S 0.0 0.1 0:00.00 hcid
2018 root 23 0 1832 368 368 S 0.0 0.1 0:00.00 sdpd
2031 root 9 -10 0 0 0 S 0.0 0.0 0:00.00 krfcommd
2075 root 25 0 12828 520 428 S 0.0 0.1 0:03.70 pcscd
2089 root 18 0 1760 448 448 S 0.0 0.1 0:00.00 acpid
2206 haldaemo 15 0 5984 1628 1308 S 0.0 0.4 0:30.82 hald
2207 root 25 0 3264 912 912 S 0.0 0.2 0:00.03 hald-runner
2216 haldaemo 25 0 2108 692 692 S 0.0 0.2 0:00.00 hald-addon-acpi
2222 haldaemo 25 0 2108 688 688 S 0.0 0.2 0:00.00 hald-addon-keyb
2235 root 16 0 2060 608 560 S 0.0 0.2 2:11.55 hald-addon-stor
2256 root 23 0 2008 336 336 S 0.0 0.1 0:00.00 hidd
2295 root 21 0 29424 940 832 S 0.0 0.2 0:01.50 automount
2382 root 25 0 8660 552 472 S 0.0 0.1 0:06.64 VBoxService
2405 root 15 0 7212 756 660 S 0.0 0.2 0:01.93 sshd
2421 root 18 0 2836 556 556 S 0.0 0.1 0:00.01 xinetd
2437 ntp 15 0 4512 4508 3496 S 0.0 1.2 0:00.26 ntpd
2474 root 25 0 4632 1028 1028 S 0.0 0.3 0:00.02 mysqld_safe
2524 mysql 15 0 137m 4688 2624 S 0.0 1.2 1:02.77 mysqld
2596 root 20 0 6968 1492 1420 S 0.0 0.4 0:00.55 master
2606 postfix 15 0 7092 1468 1404 S 0.0 0.4 0:00.30 qmgr
2611 root 18 0 2000 236 236 S 0.0 0.1 0:00.00 gpm
2625 root 15 0 25824 4340 3572 S 0.0 1.1 0:01.42 httpd
2654 root 25 0 4632 396 396 S 0.0 0.1 0:00.00 safe_asterisk
2666 asterisk 15 0 49644 5020 3580 S 0.0 1.3 1:31.91 asterisk
2672 root 18 0 5380 624 556 S 0.0 0.2 0:00.19 crond
2689 asterisk 18 0 94392 37m 3536 S 0.0 9.8 2:29.32 httpd
2690 asterisk 15 0 94908 35m 3832 S 0.0 9.6 2:29.79 httpd
2691 asterisk 20 0 94236 36m 3564 S 0.0 9.7 2:26.74 httpd
2692 asterisk 22 0 94908 36m 3548 S 0.0 9.7 2:29.30 httpd
2693 asterisk 18 0 94072 36m 3548 S 0.0 9.7 2:29.08 httpd
2694 asterisk 18 0 94452 37m 3836 S 0.0 9.9 2:27.76 httpd
2695 asterisk 21 0 95228 38m 3536 S 0.0 10.1 2:27.96 httpd
2696 asterisk 21 0 94332 37m 3544 S 0.0 9.9 2:28.77 httpd
2704 xfs 18 0 3400 576 544 S 0.0 0.1 0:00.00 xfs
2730 root 18 0 110m 2808 1580 S 0.0 0.7 0:25.48 fail2ban-server
2732 root 15 0 2768 1052 896 S 0.0 0.3 0:12.77 gam_server
2801 root 24 0 4628 1008 1008 S 0.0 0.3 0:00.01 startup.sh
2822 root 24 0 4632 1020 1020 S 0.0 0.3 0:00.01 bash
2838 root 18 0 2360 352 312 S 0.0 0.1 0:00.00 atd
2841 root 19 0 255m 5052 1268 S 0.0 1.3 0:10.71 java
2863 root 24 0 3068 508 508 S 0.0 0.1 0:00.00 incrond
2890 nobody 18 0 1916 316 292 S 0.0 0.1 0:00.02 dnsmasq
2995 root 34 19 25904 3576 2116 S 0.0 0.9 0:00.33 yum-updatesd
3039 root 15 0 3612 252 236 S 0.0 0.1 0:00.00 smartd
3042 root 17 0 1748 396 396 S 0.0 0.1 0:00.01 mingetty
3043 root 18 0 1748 376 376 S 0.0 0.1 0:00.00 mingetty
3044 root 18 0 1748 396 396 S 0.0 0.1 0:00.00 mingetty
3045 root 19 0 1748 376 376 S 0.0 0.1 0:00.00 mingetty
3046 root 20 0 1748 396 396 S 0.0 0.1 0:00.00 mingetty
3047 root 21 0 1748 376 376 S 0.0 0.1 0:00.00 mingetty
7043 postfix 18 0 7032 1816 1464 S 0.0 0.5 0:00.07 pickup
9716 asterisk 18 0 41876 19m 3672 S 0.0 5.3 0:04.88 httpd
12517 root 15 0 2408 996 784 R 0.0 0.3 0:00.01 top
16424 root 15 0 10200 2452 2228 R 0.0 0.6 0:00.48 sshd
16431 root 15 0 4636 1308 1148 S 0.0 0.3 0:00.09 bash


Another bit

crogialli's picture

The bug is clearly related to the use of the System Administration GUI.
In a night, keeping any Management window carefully closed, the system didn't increase memory usage.


What "System Administration

p_lindheimer's picture

What "System Administration GUI" display page did you have opened that you observed this?

If you can create a 'reasonably reproducible" situation please feel free to file a bug so one of the developers can have a closer look.

Please make sure to provide details of FreePBX, apache, php, etc. versions that are being run in the ticket.


__________________

Philippe Lindheimer - FreePBX Project Leader
FreePBX Training Opportunities - Click Here
Get Official Paid Support - Click Here


Status Page

crogialli's picture

The browser was open on the FreePBX auto-reloading status page.
So - when left unadvertently open- it reloads at least one hundred times in a night.
This is the reason why IMHO it is a reasonable suspect, because the page auto-generates http sessions.

Thanks

Carlo


not sure...

VoIPTek's picture

One of my initial hunches was that keeping the status report open was possibly using up the memory, however on boxes that I was looking at frequently when I left the management page in the evening and opened it the next day memory DID increase buy a good jump.

With all the recent updates I have seen an improvement, memory is not disappearing as quickly. I will leave my servers status screen open today and see if we can confirm that one way or another.

crogialli, I see your current and I think at the moment that's about all you can do. I would also say stay out of the status or for that matter the entire GUI to see if that improves the situation.

Are you running a real server or are you running a VPS?


Running in a VPS

crogialli's picture

It's in a Virtual Machine, over a Mac OSX server.
Until it's fixed, I'll try to stay out the gui and I think I'll schedule a machine restart every night at 3:00 am.


dont need to restart

VoIPTek's picture

you don't need to restart the whole system, simply restart apache and the memory clears up.


I'll follow your suggestion

crogialli's picture

Thanks VoIPTek, sure I will follow your advice.

Carlo


Philippe

VoIPTek's picture

When this was first opened memory was disappearing at an alarming rate to the point that you would have to reboot apache daily. After all the updates from when this was originally opened, I'm not sure the problem still exists.

The screen I believe that people run is the FreePBX System status which also happens to tell you the percentage of memory.

In my testing today keeping the GUI status open I do see I went from 20-22% but thats nothing, I would literally run out of memory within a day, so that has improved with all the updates. Now it may not be related to the GUI because I did have the problems as stated previously.

I don't believe the problem is there at the level it was before because I could duplicate it on 6 different servers and right now I really cant say I can duplicate it.

crogialli, run yum check-update from the command line of the server, does it tell you that you need any updates? I know you ran the scripts already and should be current, this is just a way to double check. Also after your updates have you rebooted?