Tuesday, April 26, 2016

Increase the number of open files limit on Ubuntu 12.04

On Ubuntu 12.04, every process could only use up to 1024 open files including socket handlers, file handlers, etc.
When we develop scalable programs, this could hinder the throughput of your programs. Many people try increasing the number, but apparently it is not straightforward.

I am listing the steps which work for me here:
(1) change /etc/security/limits.conf by adding the following lines:
your-user-name soft nofile 4096
your-user-name hard nofile 4096

(2) change /etc/pam.d/common-session* by adding the following line:
session required pam_limits.so

(3) logout and login again if you use ssh.



References

http://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user

Tuesday, April 12, 2016

Memory leak issue in MySQL C++ connector 1.1.7

I have used the official C++ connector (Version 1.1.7) for MySQL recently, but found some weird memory leak, though I was totally following the official documents. The connector could be found here.

How does the memory leak look like?

I used Valgrind to detect memory leak of my program, and found the memory used by the MySQL thread was not released somehow:
==20882== 8,000 bytes in 40 blocks are definitely lost in loss record 1,707 of 1,791
==20882==    at 0x4C29DB4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==20882==    by 0x620F83E: my_thread_init (in /usr/lib/libmysqlcppconn.so.7.1.1.7)

==20882==    by 0x61F7384: mysql_server_init (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==20882==    by 0x61FD1C6: mysql_init (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==20882==    by 0x61F21F3: sql::mysql::NativeAPI::LibmysqlStaticProxy::init(st_mysql*) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==20882==    by 0x61F369E: sql::mysql::NativeAPI::MySQL_NativeConnectionWrapper::MySQL_NativeConnectionWrapper(boost::shared_ptr) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==20882==    by 0x61F341D: sql::mysql::NativeAPI::MySQL_NativeDriverWrapper::conn_init() (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==20882==    by 0x61ABE63: sql::mysql::MySQL_Driver::connect(sql::SQLString const&, sql::SQLString const&, sql::SQLString const&) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)

Why does it happen?

After some investigation, I found the post in the Reference. It seems that the official document did not tell users to do anything for the MySQL_Driver pointer (e.g., this sample code), but actually it is necessary in order to release the memory allocated for the thread used by the MySQL connector.
My finalized codes look like:
    bool runQueryWithResult(const std::string &query,
            std::function callbackFunction)
    {
        sql::mysql::MySQL_Driver *sqlDriver=NULL;
        sql::Connection *connection=NULL;
        sql::Statement *stmt=NULL;
        sql::ResultSet *res=NULL;
        try
        {
            // get a driver
            sqlDriver = sql::mysql::get_driver_instance();
            // create connection
            connection=sqlDriver->connect(m_db_tcpAddress, m_db_username, m_db_password);
            // run a statement
            stmt=connection->createStatement();
            res=stmt->executeQuery(query);
            bool returnBool=false;
            callbackFunction(res, returnBool);
            delete res;
            res=NULL;
            delete stmt;
            stmt=NULL;
            connection->close();
            delete connection;
            connection=NULL;
            // this step is necessary to avoid memory leak, though it is not mentioned in the document
            sqlDriver->threadEnd();

            sqlDriver=NULL;
            return returnBool;
        }
        catch (sql::SQLException &e)
        {
            std::string warning="SQLException in "+std::string(__FILE__)+"("+std::string(__FUNCTION__)+") on line "+std::to_string(__LINE__)
                +"\n# ERR: "+std::string(e.what())
                +" (MySQL error code: "+std::to_string(e.getErrorCode())+", SQLState: "+e.getSQLState()+") with query: "+query;
            Tools::error(warning);
            if (sqlDriver!=NULL)
                sqlDriver->threadEnd();

            if (res!=NULL)
                delete res;
            if (stmt!=NULL)
                delete stmt;
            if (connection!=NULL)
            {
                connection->close();
                delete connection;
            }
            return false;
        }
    }




Reference

(1) Post about memory leak in MySQL C++ connector:
http://stackoverflow.com/questions/13082389/memory-leak-in-mysql-c-connector

(2) Official document of MySQL C++ connector:
https://dev.mysql.com/doc/connector-cpp/en/connector-cpp-examples-connecting.html