Posted on Jun 23 • Edited on Jul 5

Introducing curlev: a highly-efficient asynchronous HTTP client

I recently faced a significant challenge: making a large number of asynchronous HTTP calls to various servers, without waiting for responses but storing the data as it arrived. These requests originated from different parts of the program and were posted at various times.

After evaluating several libraries (Async++ CURL, cpr, curl-multi-asio, curlcpp, restclient-cpp...), I found that none fully met my requirements. While cpr offered the best performance, its CPU usage was too high for me.

So, I developed my own solution: curlev. It's built upon libcurl's multi-interface and libuv's asynchronous I/O capabilities. curlev achieves performance very close to cpr, but with four times less CPU consumption (and half as much memory).

Config	Time	CPU	RSS	s·CPU·MB
curlev	2.162 s	109%	14'580 KB	34
cpr	1.722 s	501%	22'016 KB	185
asyncpp-curl	4.101 s	126%	335'744 KB	1694
curlcpp	45.347 s	99%	625'204 KB	27410
curl-multi-asio	100.102 s	99%	538'164 KB	52083
liblifthttp	104.029 s	99%	528'128 KB	53116

curlev offers three modes:

A synchronous mode:

 auto http = HTTP::create( global::async ); auto code = http->GET( "https://api.example.com/get" ) .exec() .get_code();

An asynchronous mode (also available with std::future):

 auto http = HTTP::create( global::async ); http->GET( "https://api.example.com/get" ) .start(); ... ... auto code = http->join().get_code();

A detached mode, with callback:

 { HTTP::create( global::async ) ->GET( "https://api.example.com/get" ) .start( []( const auto & http ) { auto code = http.get_code(); } ); }

A std::shared_ptr is returned by the factory function HTTP::create, to ensure that the connection stays alive during the operation, even if the returned value goes out of scope.

It also provides:

all standard HTTP methods (GET, POST, PUT, PATCH, DELETE)
query parameters, form data, MIME handling, and raw bodies
custom headers and authentication
received headers and body

Here are 3 examples inspired from real usages:

auto http = HTTP::create( m_async ); http->POST( opr_callback_url ) .add_headers( { { "X-Response-ID", response_id } } ) .start( [ reference_id ]( const auto & http ) { if ( http.get_code() == 200 ) ack_notification( reference_id ); else log_warning( "opr cb failed: " + std::to_string( http.get_code() ) + ", ref=" + reference_id ); } );

auto http = HTTP::create( m_async ); auto code = http->POST( m_opr_server + "/user/notify" + user_id ) { { "message_id", "account.created" }, { "data" , data } } ) .exec() .get_code(); // if ( code == 200 && http->get_content_type() == "application/json" ) json = http->get_body(); else return false;

auto http = HTTP::create( m_async ); auto code = http->GET( m_opr_server + "/user/" + user_id ) .authentication( "mode=bearer,secret=" + token ) .exec() .get_code();