It is necessary in some cases to sync time among the nodes. This expriment uses RPC to implement network time synchronization.
RPC, Time Synchronization, Go
There are many cases where time synchonization is more or less significant. For some competitive online games, there would be a mess if various clients hold inconsistant time. In this expriment, we try to design and realize a network time syncronization in the architecture of C/S.
We have a plenty of communication methods between server and clients to choose from. Considering convenient and scalability, and finally RPC is selected. Mature RPC fragments have the advantages of service discovery, retry on failure, load balance which is rather helpful. As to the develop language, go is choosen since it is designed for such high concurrency distributed appliations.
There are several requirements from this network time syncronization application. First of all, it should be language independent and cross-platform which means the server is able to support different kinds of clients running in various platforms. Secondly, the server can serve a quantity of clients at the same time. Thirdly, in this application, only authorized clients could get time from server. Except for the requirements above, we have more things to take into consideration such as network delay, network failure, scalability, etc.
After research, we found some available packages of RPC in go language and after comperison gRPC turned out to be the best choice from our application.
It is rather common for an online application that the network latency is too high to be ignored and even package lose. This problem is more obvious for our network time synchonization application. However, it is not easy to solve this problem since the network changes unpredictedly all the time. In our expriment, we don’t want to have a extremely precise result and can tolerate a second level of error.
To do this, we record the local time before and after making RPC call which we call
stop, and obviously
stop - start is the time need for one RPC call. We assume that processing time in both sides can be ignored and the network changes little during the request. So we can calculate the time to reach another side simply by dividing the difference by 2.
start := time.Now().UnixNano()
Then, we can get the real time by subtracting delay from the time replied.
realtime = reply.time - delay
Protobuf is a serialization method and is adopted by gRPC in message exchange.
syntax = "proto3";
Here we generates two pb files since we will also create a python client as well.
Get gRPC package and protobuf-go plugin package
go get -u google.golang.org/grpc
Get gRPC package and protobuf plugin-python package
sudo apt install python-pip
Although we can pass token every time we make RPC call, it is not a good design. Luckily, gRPC provides the ability to verify clients (tls). We generates a key file along with a certificate file, RPC call from unauthorized clients without valid certificate file will be rejected by the gRPC fragment automatically. What’t more, gRPC can also distinguish each client by setting a unique token, we skipped the token verification since the certificate files is secure enough for our case.
With the terribe network environment where full of monitors and sniffers, it is significant to encrypt messages in network. After research, we find gRPC would encrypt the messages if tls is used, thus we don’t have to trouble ourselves.
Now, we issue certificate files to
ntp.newnius.com and use them for authorization later.
openssl req -x509 -newkey rsa:4096 -keyout server-key.pem -out server-cert.pem -days 365 -nodes -subj '/CN=ntp.newnius.com'
We realized two version clients in go and python.
The client works as follows:
- Start a new thread to make RPC calls in a certain frequency and update the global variable
delaywhich means the time delay between server and client (network delay included)
- Main thread displays current time every second. The outputed time is calculated by
now = localtime + delay
Notice that we didn’t display time in digital mode or analogue mode since they are not the main purpose for this expriment and GUI programming is so time-consuming. Instead, we just print current time every second.
# -*- coding: utf-8 -*-
Several tests were made for different purposes.
We deployed the server in a remote VPS whose average
ping latency from the develop machine is around 400ms and made following settings in server / client:
- Add the time replied by server by 100 seconds to simulate a situation that the client is 100s behind server
- Sleep for 5s before & after RPC call to generate a longer network delay
The result shows that the client successfully calculated the real time fixed by delay and retry on failure.
Since we don’t have Windows system in hand, so all the work were made in Linux. But consider that python is a cross-platform language so we can say that the client can run on most platforms.
In our tests, only clients using the right method and certificate files can get response from server.
We deployed 6 VPSs to perform as clients and one as server. The resource of VPSs ranges from 1 Core/0.5G Ram to 1 Core/2G Ram, and the server is deployed in a 1 core/2G Ram VPS. All of them are in system Linux without desktop and most of the resources are vacant. To better simulate the real environment, the VPSs are located around the world.
First of all, we tried to run as many as possible clients in the 6 nodes, each client would send a query requst every 5 seconds. The total number of client is about 2000.
After that, we run a client in our develop machine, and the log showed that the RPC call is made successfully. The figures below shows that under the concurrency of 1000, the server still works perfect and can hold more clients.
There are still a lot of work to do with this application.
Although the communication delay is considered, it is not that precise for higher requirements. To achieve this goal, we can apply more approaches to fix this with more than one query. Secondly, in linux system, each process can only have at most 1024 active connections, which means only 1024 clients can be served as the same time. What’s more, each connection may consume more resource than needed. These limitations can be solved by changing the settings.
What’s more, gRPC uses
HTTP, which is based on
TCP in communication, this improves performance for frequent queries. But in our case, the request may be very slow and failure is acceptable, so we can use UDP to serve more clients.
This expriment shows that RPC is a good fragment to make communications between server and client. With the help of a muture fragement, we can easily build a high available online application.
 Go RPC 开发指南