Hire a web Developer and Designer to upgrade and boost your online presence with cutting edge Technologies

Monday, December 5, 2022

Uploading data to a webserver – C/C++ and CURL

 this guide demonstrates how you can use a cURL library to upload data from an application written in C (or C++) to a webserver. Although at times I refer to previous examples, you can use this in any number of applications.

This will touch on a number of areas and require a little more prep work than previous examples, due to needing a few extra libraries and a server running PHP and SQL. In a nutshell, you have to cobble together a string with all your data in, send it along to your server with a request and then have that server do something with it. The steps covered are:

  1. (Setup) LAMP server and database
  2. (Client) Generate a URL string from variables in your program
  3. (Client) Send a web request to the URL generated in step 1
  4. (Server) Process the uploaded string to retrieve values from data
  5. (Server) Store the data as a new record in a database

Finally, there are some really quick and horrible things that I do here, with minimal catching of errors, but the purpose is to show the process from start to finish as briefly as possible!

Environment

First of all, its worth mentioning the set-up I am using for this system. Just to demonstrate, I have a LAMP server running on Ubuntu with a MySQL database. I won’t cover how to actually get as far as having those running because there’s plenty of articles that explain how to do that. The only other thing you may want to use is phpMyAdmin, as this makes database management via a web UI much easier than just command line interaction. It also helps hugely to understand basics of HTML/PHP – but you may well be able to stumble through this if you’ve never touched either before!

Setup

Database Table

The first thing to do is to create a new table to store the records in. This may as well just be a spreadsheet at this point, but later on I will be talking about queries and combining data from different tables. You can either create a new table with phpMyAdmin by selecting the appropriate settings from the code below or you can simply copy and paste the entire thing into an SQL query box and run it, which will create a new table for you.

CREATE TABLE inventory
(
hostName varchar(25) NOT NULL,
hostIP varchar(25),
hostMAC varchar(18),
hostCreateDate timestamp DEFAULT CURRENT_TIMESTAMP,
vlanID int(4),
switchPort varchar(32),
switchIP varchar(25),
switchName varchar(64),
switchPlatform varchar(64),
UNIQUE (hostName)
);

This single table consists of host information (name, mac, ip), switch information (name, platform, IP, port) and a date that the entry was created. Note that this also adds the constraint to the hostName to ensure that it is unique (and stipulates that it can’t be empty). There is a logical argument that, perhaps, it should be the MAC address that is unique; but I am quite happy with my domain and that it won’t give me hostname duplication for this demo. Plus, the focus is on getting data of different types in, rather than necessarily what the data is.

PHP page to process data

This is the part that is going to bring it all together later. When we want to place data into the database, we need something that will take data sent to the server and process that into the database. This file is going to store the database credentials, so its really important that it isn’t made available outside of your webserver. Since PHP files execute on the server, rather than the client, the file contents aren’t sent back when requested (unless the PHP service stops and breaks, in which case everything will be displayed in plaintext..). In fact, we really don’t have to – or necessarily want to – output anything with a PHP file; in this instance, it is simply used as a way to process data from clients to be stored in a database as follows:

The first thing to do is to specify some variables for the database details. These are used to connect to the database (on the same server, in this case):

<?php
	$servername = "localhost";
	$username = "user";
	$password = "Passw0rd";
	$dbname = "inventory";

//Rest of the code goes here

?>

 

The next thing to do is to create a connection object that handles the connection to a MySQL database, passing it the above variables as parameters:

$conn = new mysqli($servername, $username, $password, $dbname);

Before we can go any further, however, we need to actually have some data to process – or know what data we want to process. I’ll come back to this in a while, but first let’s generate the URL string.

Generate a URL string

At this point, I am making an assumption that you already have variables that you want to send. The key thing is to prepare that data to be sent to the server.

There are a number of ways in which you could send data with a web request; HTTP GET and POST being probably the most suited.

  • GET is easy; you can just construct a string consisting of the URL of the server to send the data too and then append all the extra data you want onto the end of it. However, you’re limited to 2048 characters, the URL is very “visible” (like any URL, the data included is going to pop up in search history, it can be cached) and so it is better used when trying to retrieve data (IE when you need to include some specific parameters as part of the request).
  • POST, on the other hand, hasn’t got the same limits on what – and how much – data you can include in the request. Possibly more crucially, however, is that the data isn’t sent along in the header of the request, so it isn’t as easily seen directly through the URL.

For good practice, I’m going to focus on using POST to upload the data. We’re using the request to send data, not simply retrieve it (which is where using GET would be appropriate). We can upload quite a bit of data this way, but it has to be formatted first. The string is pretty much the same for GET as it would be for POST; the difference is that the POST data will be sent separately to the header, in the body of the request, in the following format:

 

variable1=value&variable2=value&variable3=value

These are sets of variables and values that are accessible by the page requested; but the variable name must be used as it is on the page. If your local variable is myHostName, for example, but you send it as val1, then once it has reached the server, it has to be referred to by the name val1.

So how do you construct this string from many smaller strings?

There are a few ways to do this (str::append, for example). I’m using a mixture of C and C++ throughout these examples (which is probably a horrible thing to do in practice), but to break down how it works, I am going to manually stick the different strings together.

First of all, I’ve created a function called generatePOSTData() that will prepare the different parts of a string to be added together, consisting of 8 example variables. Note that each one has an equals operator next to it, which is used by the webserver:

void generatePOSTData()
{
	char *prefixhost = "host=";
	char *prefixip = "ip=";
	char *prefixmac = "mac=";
	char *prefixvlan = "vlan=";
	char *prefixswPort = "swPort=";
	char *prefixswName = "swName=";
	char *prefixswIP = "swIP=";
	char *prefixswMAC = "swMAC=";

	
	int slhost = strlen(prefixhost);
	int slip = strlen(prefixip);
	int slvlan = strlen(prefixvlan);
	int slmac = strlen(prefixmac);
	int slswPort = strlen(prefixswPort);
	int slswName = strlen(prefixswName);
	int slswIP = strlen(prefixswIP);
	int slswMAC = strlen(prefixswMAC);
...

Note that this also calculates the length of each of the strings we’ve just created. This is important when adding the strings together, later.

Next, we can take the strings and add them together, storing them in another variable. The function, addStrings, is something I have discussed in a previous post.

	addStrings(&host, prefixhost, systemhostname, slhost, slsystemhostname);
	addStrings(&ip, prefixip, systemip, slip, slsystemip);
	addStrings(&vlan, prefixvlan, systemvlan, slvlan, slsystemvlan);
	addStrings(&mac, prefixmac, systemmac, slmac, slsystemmac);

	addStrings(&swPort, prefixswPort, systemswitchport, slswPort, slsystemswitchport);
	addStrings(&swIP, prefixswIP, systemswIP, slswIP, slsystemswIP);
	addStrings(&swName, prefixswName, systemswName, slswName, slsystemswName);
	addStrings(&swMAC, prefixswMAC, systemswMAC, slswMAC, slsystemswMAC);
}

void addStrings(char ** result, const char * prefix, const char * body, int &prefixlength, int &bodylength)
{
	*result = (char *)malloc((prefixlength + bodylength + 1) * sizeof(char));
	memcpy(*result, prefix, prefixlength);
	memcpy(*result + prefixlength, body, bodylength);
	(*result)[prefixlength + bodylength] = '\0'; //Must be set like this as array notation takes precendence over a dereferencing
	prefixlength = strlen(*result);
}

Finally, we can start to construct a single string of all of these together.

requestString = (char *)malloc((slhost + slip + slswPort + slvlan + slmac + slswName + slswIP + slswMAC + 7 + 1) * sizeof(char));

This string is going to be long enough for each of the lengths of the above variables, plus one extra character for the ampersands to connect them. Plus a terminating character.

Now it is time to go through each string (I could use a recursive function here, but I’m keeping it long and simple here.. optimisation will come later!) and, bit by bit, copy the next piece of data over from individual variables over to requestString with an ampersand added after each set of values:

memcpy(requestString, host, slhost);
	requestString[slhost] = '&'; 

	memcpy(requestString+ slhost + 1,
		ip, slip);
	requestString[slhost + slip + 1] = '&';

	memcpy(requestString+ slhost + slip + 2,
		swPort, slswPort);
	requestString[slhost + slip + slswPort + 2] = '&';

	memcpy(requestString+ slhost + slip + slswPort + 3,
		vlan, slvlan);
	requestString[slhost + slip + slswPort + slvlan + 3] = '&';

	memcpy(requestString+ slhost + slip + slswPort + slvlan + 4,
		mac, slmac);
	requestString[slhost + slip + slswPort + slvlan + slmac + 4] = '&';

	memcpy(requestString+ slhost + slip + slswPort + slvlan + slmac + 5,
		swName, slswName);
	requestString[slhost + slip + slswPort + slvlan + slmac + slswName + 5] = '&';

	memcpy(requestString+ slhost + slip + slswPort + slvlan + slmac + slswName + 6,
		swIP, slswIP);
	requestString[slhost + slip + slswPort + slvlan + slmac + slswName + slswIP + 6] = '&';

	memcpy(requestString+ slhost + slip + slswPort + slvlan + slmac + slswName + slswIP + 7,
		swMAC, slswMAC);
	requestString[slhost + slip + slswPort + slvlan + slmac + slswName + slswIP + slswMAC + 7] = '\0';

This is kind of ridiculous, admittedly. I could just use the addStrings function from above, or as I said before, just functions in C++ to do this. However, it shows you how you really have to add strings together at a lower level than simply doing something at a higher level such as string1 += string2.

Send a web request with cURL to the webserver

Now that we have the post string, we can prepare it for sending with cURL; and it isn’t very hard. The first thing to do is to create a new curl object and then initialise it. If that works, you can use curl_setopt to set the URL on the object to your webpage (so http://www. your-site.com/ sendData.php) and the post string to send that was constructed above. Finally, you can initiate the request with curl_easy_perform and then do cleanup to remove the cURL object.

//Library available from https://curl.haxx.se/libcurl/
#define CURL_STATICLIB //You may or may not need this!
#include "curl/curl.h"
#pragma comment ( lib, "libcurl.lib" )

//...
//...

	CURL * curl;
	curl_global_init(CURL_GLOBAL_ALL);
	CURLcode res;
	curl = curl_easy_init();
	if (curl)
	{
		curl_easy_setopt(curl, CURLOPT_URL, address);
		curl_easy_setopt(curl, CURLOPT_POSTFIELDS, requestString);			
		res = curl_easy_perform(curl);
		if (res != CURLE_OK)
		{
			fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
		}	
		curl_easy_cleanup(curl);
	}
	curl_global_cleanup();

Outside the scope of this document would be handling the response from the server, but it is important to provision for this later as this will allow you to know whether or not things worked on the client side!

Process the uploaded string

Now it is time to go back to where we were with the PHP page. We know what data is going to be sent now, so we know what will be received by the page.

Loading in the data

Although the string was all wrapped up with ampersands and formatted with a header, when the page receives the data, it is available as an associative array called $_POST. This means that you can access a value by using the name of it in the index (for example, $_POST[“host”] will return you the value of the hostname). The other thing about $_POST is that it is accessible from anywhere on your page, as it is a superglobal type. However, in this example, we only access it once to load in the values and process them into variables used throughout the page.

Where you declare the other variables in your code, so around $conn or the database details, add the following:

$host = mysql_escape_string($_POST['host']);
$ip = mysql_escape_string($_POST['ip']);
$swPort = mysql_escape_string($_POST['swPort']);
$swIP = mysql_escape_string($_POST['swIP']);
$swName = mysql_escape_string($_POST['swName']);
$swMAC = mysql_escape_string($_POST['swMAC']);
$vlan = mysql_escape_string($_POST['vlan']);
$mac = mysql_escape_string($_POST['mac']);

What this will do is quite important; it (hopefully) will prevent against SQL injection attacks by escaping the string. In other words, it stops code from being added to text that could unexpectedly end the string and roll over into executing the code that was submitted. However, see this post for a bit more information – and why you might not want to use it in the real world.

Additionally, we now can refer to the variables by their name prefixed with a dollar symbol, rather than having to use $POST[‘variableName’] for everything. (Interestingly there are some differences between single and double quotes in php.).

Validate the data

The next test we want to do on the incoming data is to make sure it is, in fact, populated and not simply. This can be done by simply checking through each variable and seeing if it is null or not. Note that anything echo’d will be returned to the requestor. If this was a browser, you would see the response on screen, but this application doesn’t have a way to handle this, yet.

if ($host == NULL)
	{
		echo "Error, host is empty! ";	
	}

There is also one more thing that I want to add in; a test to see if the host is valid or not. If there is junk data and there are blank entries, for fields that must not be empty, this can create problems for us. If we specify an invalidation flag, that can be set if one of the variables is empty, then we can stop any database processing from happening.

$testInValid = 0;

if ($host == NULL)
	{
		echo "Error, host is empty! ";
		$testInValid = 1;
	}

if ($ip == NULL)
	{
		echo "Error, ip is empty! ";		
		$testInValid = 1;
	}

if ($swPort == NULL)
	{
		echo "Error, swPort is empty! ";
		$testInValid = 1;
	}

if ($vlan == NULL)
	{
		echo "Error, vlan is empty! ";		
		$testInValid = 1;
	}

if ($mac == NULL)
	{
		echo "Error, mac is empty! ";		
		$testInValid = 1;
	}

if ($swName == NULL)
	{
		echo "Error, swName is empty! ";		
		//$testInValid = 1;
	}

if ($swMAC == NULL)
	{
		echo "Error, swMAC is empty! ";		
		//$testInValid = 1;

	}

if ($swIP == NULL)
	{
		echo "Error, swIP is empty! ";
		//$testInValid = 1;
	}

if (testInValid == 0)
{

We begin by declaring a variable called testInValid which, by default, will remain as 0 so long as all the key pieces of data exist. At the bottom, we continue the program so long as testInValid remains as 0. It is a quick and dirty trick, but it ensures that we don’t execute code if it could break anything in the database.

Store the data as a new record in a database

With the data validated and in the form, it is time to add it to the database. Be sure that you have you added the following line to create the connection object:

$conn = new mysqli($servername, $username, $password, $dbname);

The first thing to do is to see if we can connect to it. This will also catch the error if it can’t, but you don’t have to populate it yet:

if ($conn->connect_error)
{
}

Instead, we can now create an SQL Select statement, which will be run as a query by the object.

$sqlSelect = "SELECT * from hosts WHERE hostName = '$host'";

What this will do, is to return records for any hosts that exist with the hostname that we have specified in the request. Note that, although the query is in double quotation marks, the hostname is within single quotes. This allows us to use the $host variable from the page, expanded to its actual value. Another way to do that would be to use a full stop to join two strings together, with the second string being $host and the first string being the SQL statement up until the end of the equals sign, as so:

$sqlSelect = "SELECT * from hosts WHERE hostName = " . $host;

Back to the statements: we need another one. This one will be the SQL Insert statement, used to add an entry into the database.

$sqlInsert = "INSERT INTO hosts (hostName, hostIP, switchPort, switchIP, switchName, switchPlatform, vlanID, hostMac)		

				VALUES ('$host', '$ip', '$swPort', '$swIP','$swName','$swMAC','$vlan','$mac')";

 

Finally, we run the queries. First we check to see if there is already a record with that hostname which – if there is – means that we do nothing other than report that to be the case. However, if there are no results for the query, this means that there are no hosts with that name and we can now run the second query. If that returns true – in other words, there were no errors – then we can assume it worked and output that to be the case.

$result = $conn->query($sqlSelect);

if ($result->num_rows > 0)
{
 echo "Host already exists!"
}

else
if ($conn->query($sqlInsert) === TRUE) 
{
	echo "Success!";
}

Viewing the results

You can now check this by making a new page with the same database connection details that fetches and outputs the data to a webpage. Below is a commented example of something that you can use to view the results into a table. Note that anything echo’d will be output as HTML – you can view how the code is generated in the browser by right clicking the screen (in most browsers) and clicking “View Source” when you run this:

<?php

//Connection strings
	$servername = "myserver";
	$username = "user1";
	$password = "p@ssw0rd";
	$dbname = "database1";
	$tableName = "inventory";

//Connection object
	$conn = new mysqli($servername, $username, $password, $dbname);

//Check to see if the connection succeeds
	if ($conn->connect_error) {
		die("Connection failed: " . $conn->connect_error);
	}


//SQL statement to select various fields from the table. Note that you could say, instead, "SELECT * FROM" . $tableName;	
	$sql = "SELECT hostName, hostIP, hostMac, hostCreateDate, vlanID, switchPort, switchIP FROM ". $tableName ;

//Run the query and store the results in an array
	$result = $conn->query($sql);

	if ($result->num_rows > 0) 
	{	
//This will spit out a table with some headers on the first row
		echo "<table><tr><th>Hostname</th><th>IP Address</th><th>MAC</th><th>Switch IP</th><th>Switchport ID</th><th>VLAN ID</th><th>Created On</th></tr>";

//Here we just create a new table row with the results displayed in each column
		while($row = $result->fetch_assoc()) 
		{
			  echo "<tr><td>".
			  $row["hostName"]."</td><td>". 
			  $row["hostIP"]."</td><td>". 
			  $row["hostMac"]."</td><td>". 
			  $row["switchIP"]."</td><td>" .
			  $row["switchPort"]."</td><td>" .
			  $row["vlanID"]."</td><td>".
			  $row["hostCreateDate"]."</tr>"; 
		}
	}

//Just in case we have nothing!
	else 
	{
		echo "0 results";
	}

$conn->close();

?>

 

That concludes a very hacky – but functional – guide to how to get data from an application written in C/C++ into a database and displayed on a webpage using cURL. Please do post comments, issues and questions in reply!

 

I demonstrated an example of uploading data to a webserver with cURL. The thing that bugged me was, however, that this used external libraries which can be annoying to deploy with. One option is to statically link the cURL libraries, which will make it huge, but I decided to go native in Windows and give Winsock a go.

Working from the example in Part 1, one of the first things I decided to do was to remove any reference to cURL. Actually, the only place you need to really do this in the previous example would be in the function where you actually upload the string (and, of course, any library or header reference). With that out of the way, we now need to make a socket (which, on Windows, you should only need to use winsock.h and Ws2_32.lib to achieve).

WSADATA wsaData; 
if (WSAStartup(MAKEWORD(2, 0), &wsaData) != 0)
{
	logfileOutput("WSAStartup failed.");
	exit(1);
}

This first snippet is needed to initialise Winsock (you might want to also add a way to capture the exit code, too). Once that’s done, I’ve gone and replaced the old cURL code with some Winsock code:

struct addrinfo hints;
	ZeroMemory(&hints, sizeof(hints));
	hints.ai_family = AF_INET;          // IPv4
	hints.ai_protocol = IPPROTO_TCP;    // TCP
	hints.ai_socktype = SOCK_STREAM;    

	struct addrinfo* targetAdressInfo = NULL;
	DWORD getAddrRes = getaddrinfo(address, NULL, &hints, &targetAdressInfo);

	if (getAddrRes != 0 || targetAdressInfo == NULL)
	{
		logfileOutput("Could not resolve the hostname.");
	}

	SOCKADDR_IN sockAddr;
	sockAddr.sin_addr = ((struct sockaddr_in*) targetAdressInfo->ai_addr)->sin_addr;    
	sockAddr.sin_family = AF_INET;  // IPv4
	sockAddr.sin_port = htons(80);  // HTTP Port: 80
								
	freeaddrinfo(targetAdressInfo);

	SOCKET webSocket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
	if (webSocket == INVALID_SOCKET)
	{
		logfileOutput("Creation of the socket failed!");
	}

	printf("\nConnecting... ");
	if (connect(webSocket, (SOCKADDR*)&sockAddr, sizeof(sockAddr)) != 0)
	{
		logfileOutput("Could not connect");
		closesocket(webSocket);
	}
	printf("Connected");


	// Here's where we send the data
	const char* httpRequest = postdata;
	int sentBytes = send(webSocket, httpRequest, strlen(httpRequest), 0);
	if (sentBytes < strlen(httpRequest) || sentBytes == SOCKET_ERROR)
	{
		logfileOutput("Could not send the request to the server");
		closesocket(webSocket);
	}

 

Because it is an incredibly quick and dirty example, I won’t explain it too much. I would actually prefer using something like cURL, which has already been done, but the basic idea is that you set up a socket, connect to the webserver and then send the data. Notice that there is no provision for receiving or interpreting responses, so this is really unhelpful for debugging. But the point is to illustrate that you can send data using what is already provided by Microsoft in the Winsock library.

The two piece of data – httprequest (and data) and address – represent the webserver URL (e.g. http://10.0.0.1/) and the request data itself. This data is not simply the string that we had previously; we need to make another string that wraps around the string of values to send. To do that, I’ve made the following code, which creates a string in the format that is expected for the data to be received in:

	sprintf_s(postdata,
		"POST %s HTTP/1.1\r\n"
		"Host: %s\r\n"
		"Content-Type: application/x-www-form-urlencoded\r\n"
		"Content-Length: %i\r\n\r\n"
		"%s\r\n", settingsPage, settingsServer, strlen(requestString), requestString);

To break it down a little:

  • The first part of this, POST %s, tells the page defined by settingsPage that this is POST data. The page should be the one that you created earlier; so this could be senddata.php or something.
  • It is important for the content-type field to report that it is application/x-www-form-urlencoded and not text/html as I originally had tried to do. It just won’t send the data in the body as you want it to, otherwise!
  • Content-length has to be correct – but that is simply the length of the requestString.
  • Finally, after two carriage returns (and I think that this has to be the case), you include the data you want to include (in other words, the payload of requestString) that was generated in the previous example.

You can test what it comes out as with a simple printf of postdata, which should show you what the server will see. This is then sent to the server in the socket code above.

Again, this is a very quick and dirty way to get a socket to send the same data as in the last example and is really just to show that you can do it. There is very little error catching, format checking and it doesn’t capture a response. But with a bit of work, it can eliminate a lot of un-necessary code that more cumbersome (cURL) libraries might include if all you’re looking to do is to create the bare minimum for a small footprint binary executable.

 

Posted by H at 4:11 AM
Labels: , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)