Opened 7 years ago

Closed 7 years ago

#136 closed defect (fixed)

Infinite loop while cleaning up GnuTLS session

Reported by: Sunil Mohan Adapa Owned by: Daniel Kahn Gillmor
Priority: critical Component: code
Version: Keywords:
Cc:

Description

I noticed that some apache2 processes take 100% CPU after a few requests. Stack trace shows mod-gnutls as the problem. This problem causes embedded devices to slow down to an unusable state after using them for a while.

A quick read tells us that cleanup_gnutls_session() getting stuck in a loop due to GNUTLS_E_INTERRUPTED or GNUTLS_E_AGAIN being returned by gnutls_bye(). After mgs_transport_write(), gnutls understands these errnos even though errno is not set during that function?

Following is the debug information I collected. I am available to get more information or try out patches if necessary. Thanks in advance.

How to Reproduce:

The problem is easily reproducible on my machines by sending a lot of requests to Apache. Most requests don't cause a problem while some lead to the problem. I believe the problem occurs when using mod-gnutls and proxying connections. Also this is more observable on pages taking time to load.

$ ab -v 1 -c 8 -n 800 -H 'Cookie: XXX' ​https://freedombox-vm1/plinth/sys/users/

Stack traces:

I collected several stack traces by letting the process run for a while and then interrupting it again.

#0  mgs_transport_write (ptr=0x7f785b260028, buffer=<optimized out>, len=8023) at gnutls_io.c:871
#1  0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>, session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>, giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
#2  _gnutls_writev (total=8023, giovec_cnt=<optimized out>, giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
#3  _gnutls_io_write_flush (session=session@entry=0x564c16d28710) at gnutls_buffers.c:699
#4  0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710, how=how@entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
#5  0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at gnutls_hooks.c:722
#6  0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
#7  apr_pool_clear (pool=0x7f785b285028) at /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
#8  0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg@entry=7, child_bucket=child_bucket@entry=0) at prefork.c:616
#9  0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0) at prefork.c:824
#10 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:932
#11 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1128
#12 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028, plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
#13 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778

#0  write_flush (ctxt=ctxt@entry=0x7f785b260028) at gnutls_io.c:624
#1  0x00007f786419ee32 in mgs_transport_write (ptr=0x7f785b260028, buffer=<optimized out>, len=8023) at gnutls_io.c:871
#2  0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>, session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>, giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
#3  _gnutls_writev (total=8023, giovec_cnt=<optimized out>, giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
#4  _gnutls_io_write_flush (session=session@entry=0x564c16d28710) at gnutls_buffers.c:699
#5  0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710, how=how@entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
#6  0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at gnutls_hooks.c:722
#7  0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
#8  apr_pool_clear (pool=0x7f785b285028) at /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
#9  0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg@entry=7, child_bucket=child_bucket@entry=0) at prefork.c:616
#10 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0) at prefork.c:824
#11 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:932
#12 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1128
#13 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028, plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
#14 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778

#0  0x00007f78696022e0 in apr_bucket_alloc@plt () from /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
#1  0x00007f7869604591 in apr_bucket_flush_create () from /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
#2  0x00007f786419dc1c in write_flush (ctxt=ctxt@entry=0x7f785b260028) at gnutls_io.c:642
#3  0x00007f786419ee32 in mgs_transport_write (ptr=0x7f785b260028, buffer=<optimized out>, len=8023) at gnutls_io.c:871
#4  0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>, session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>, giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
#5  _gnutls_writev (total=8023, giovec_cnt=<optimized out>, giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
#6  _gnutls_io_write_flush (session=session@entry=0x564c16d28710) at gnutls_buffers.c:699
#7  0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710, how=how@entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
#8  0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at gnutls_hooks.c:722
#9  0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
#10 apr_pool_clear (pool=0x7f785b285028) at /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
#11 0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg@entry=7, child_bucket=child_bucket@entry=0) at prefork.c:616
#12 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0) at prefork.c:824
#13 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:932
#14 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1128
#15 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028, plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
#16 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778

#0  _mbuffer_head_get_next (cur=0x564c16d2c9a0, msg=msg@entry=0x7fff2ce06a10) at gnutls_mbuffers.c:191
#1  0x00007f78669d83bd in _gnutls_io_write_flush (session=session@entry=0x564c16d28710) at gnutls_buffers.c:682
#2  0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710, how=how@entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
#3  0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at gnutls_hooks.c:722
#4  0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
#5  apr_pool_clear (pool=0x7f785b285028) at /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
#6  0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg@entry=7, child_bucket=child_bucket@entry=0) at prefork.c:616
#7  0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0) at prefork.c:824
#8  0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:932
#9  prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1128
#10 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028, plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
#11 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778

#0  0x00007f7869602caa in apr_brigade_cleanup () from /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
#1  0x0000564c1555926a in ap_core_output_filter (f=0x7f785b2641e0, new_bb=0x7f785b2640e8) at core_filters.c:385
#2  0x00007f786419dc4d in write_flush (ctxt=ctxt@entry=0x7f785b260028) at gnutls_io.c:645
#3  0x00007f786419ee32 in mgs_transport_write (ptr=0x7f785b260028, buffer=<optimized out>, len=8023) at gnutls_io.c:871
#4  0x00007f78669d86d5 in _gnutls_writev_emu (session=<optimized out>, session=<optimized out>, vec=<optimized out>, giovec_cnt=<optimized out>, giovec=<optimized out>, fd=<optimized out>) at gnutls_buffers.c:447
#5  _gnutls_writev (total=8023, giovec_cnt=<optimized out>, giovec=0x7fff2ce06a20, session=0x564c16d28710) at gnutls_buffers.c:505
#6  _gnutls_io_write_flush (session=session@entry=0x564c16d28710) at gnutls_buffers.c:699
#7  0x00007f78669d4d5c in gnutls_bye (session=0x564c16d28710, how=how@entry=GNUTLS_SHUT_WR) at gnutls_record.c:281
#8  0x00007f78641a329e in cleanup_gnutls_session (data=0x7f785b260028) at gnutls_hooks.c:722
#9  0x00007f78693e1cfe in run_cleanups (cref=0x7f785b285098) at /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:2352
#10 apr_pool_clear (pool=0x7f785b285028) at /tmp/buildd/apr-1.5.2/memory/unix/apr_pools.c:762
#11 0x00007f78609ab4ce in child_main (child_num_arg=child_num_arg@entry=7, child_bucket=child_bucket@entry=0) at prefork.c:616
#12 0x00007f78609ab974 in make_child (s=0x7f7869c79470, slot=7, bucket=0) at prefork.c:824
#13 0x00007f78609ac85d in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:932
#14 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1128
#15 0x0000564c15541eae in ap_run_mpm (pconf=0x7f7869ca9028, plog=0x7f7869c76028, s=0x7f7869c79470) at mpm_common.c:94
#16 0x0000564c1553b516 in main (argc=3, argv=0x7fff2ce070c8) at main.c:778

Software versions:

root@freedombox-vm1:~# dpkg -l libapache2-mod-gnutls* apache2* libgnutls*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                            Version                      Architecture                 Description
+++-===============================================-============================-============================-===================================================================================================
ii  apache2                                         2.4.20-1                     amd64                        Apache HTTP Server
un  apache2-api-20120211                            <none>                       <none>                       (no description available)
ii  apache2-bin                                     2.4.20-1                     amd64                        Apache HTTP Server (modules and other binary files)
ii  apache2-data                                    2.4.20-1                     all                          Apache HTTP Server (common files)
ii  apache2-dbg                                     2.4.20-1                     amd64                        Apache debugging symbols
un  apache2-doc                                     <none>                       <none>                       (no description available)
un  apache2-suexec-custom                           <none>                       <none>                       (no description available)
un  apache2-suexec-pristine                         <none>                       <none>                       (no description available)
ii  apache2-utils                                   2.4.20-1                     amd64                        Apache HTTP Server (utility programs for web servers)
un  apache2.2-bin                                   <none>                       <none>                       (no description available)
un  apache2.2-common                                <none>                       <none>                       (no description available)
ii  libapache2-mod-gnutls                           0.7.4-2                      amd64                        Apache module for SSL and TLS encryption with GnuTLS
ii  libapache2-mod-gnutls-dbgsym                    0.7.4-2                      amd64                        Debug symbols for libapache2-mod-gnutls
ii  libgnutls-deb0-28:amd64                         3.3.20-1                     amd64                        GNU TLS library - main runtime library
ii  libgnutls-openssl27:amd64                       3.4.11-4                     amd64                        GNU TLS library - OpenSSL wrapper
ii  libgnutls30:amd64                               3.4.11-4                     amd64                        GNU TLS library - main runtime library
un  libgnutls30-dbg                                 <none>                       <none>                       (no description available)
ii  libgnutls30-dbgsym:amd64                        3.4.11-4                     amd64                        Debug symbols for libgnutls30

Apache MPM:

root@freedombox-vm1:~# a2query -M
prefork

Apache Site Configuration:

##
## On all sites, provide Plinth on a default path: /plinth
##
## Requires the following Apache modules to be enabled:
##   mod_headers
##   mod_proxy
##   mod_proxy_http
##
<Location /plinth>
    ProxyPass        http://127.0.0.1:8000/plinth
    ## Send the scheme from user's request to enable Plinth to redirect
    ## URLs, set cookies, set absolute URLs (if any) properly.
    RequestHeader    set X-Forwarded-Proto 'https' env=HTTPS

    ## Allow traffic only from private networks
    <RequireAny>
        ## IPv4 local addresses
        Require ip 127.0.0.0/8

        ## IPv4 link local addresses
        Require ip 169.254.0.0/16

        ## IPv4 class A private addresses
        Require ip 10.0.0.0/8

        ## IPv4 class B private addresses
        Require ip 172.16.0.0/12

        ## IPv4 class C private addresses
        Require ip 192.168.0.0/16

        ## IPv6 local address
        Require ip ::1

        ## IPv6 link local addresses
        Require ip fe80::/10

        ## IPv6 private addresses
        Require ip fc00::/7
    </RequireAny>
</Location>

Change History (2)

comment:1 Changed 7 years ago by Sunil Mohan Adapa

Forgot to mention that when compiled with --enable-maintainer, I see the following messages repeated continuously in /tmp/gnutls_debug:

<3> ASSERT: gnutls_record.c:284
<3> ASSERT: gnutls_buffers.c:462
<2> WRITE: -1 returned from 0x7f926dc0e028, errno: 11
<3> ASSERT: gnutls_record.c:284
<3> ASSERT: gnutls_buffers.c:462

comment:2 Changed 7 years ago by Sunil Mohan Adapa

Resolution: → fixed
Status: new → closed

This is now fixed in 0.7.5.

Note: See TracTickets for help on using tickets.