Published on

Part 1 - A simple gRPC service

This is Part 1 in the gRPC Tutorial Series show casing best practises and tooling in building and deploying services using gRPC (mostly in Go).

Introduction

Client-server communication is a fundamental part of modern software architectures. Clients (on various platforms - web, mobile, desktop, an even IoT devices) request functionality (data and views) that servers that compute, generate and provide. There have been several paradigms facilitating this - REST/Http, SOAP, XML-RPC and others.

gRPC is a modern, open source and highly performant remote procedure call (RPC) framework developed by Google enabling efficient communication in distributed systems. gRPC also uses an interface definition language (IDL) - protobuf - to define services, define methods, messages as well as in serializing structure data between servers and clients. Protobuf as a data serialization format is powerful and efficient - especially compared to text based formats (like Json). This makes a great choice for applications that require high performance and scalability.

A major advantage gRPC confers is its ability to generate code for several clients and servers in several languages (Java, Python, Go, C++, JS/TS) as well as targeting various platforms and frameworks. This simplifies implementing and maintaining consistent APIs (via a source-of-truth IDL) in a language and platform agnostic way. gRPC also offers features like streaming (one way and bi-directional), flow control and flexible middleware/interceptor mechanisms making it a superb choice for real-time applications and microservice architectures.

What makes the gRPC shine is its plugin facility and ecosystem for extending it on several fronts. With plugins just some of the things you can do are:

  • Generate server stubs to implement your service logic
  • Generate clients to talk to these servers
  • Target several languages (golang, python, typescript etc)
  • Even targeting several transport types (HTTP, TCP etc)

Here is an awesome list of curated plugins for the grpc ecosystem. For example - you can even generate an http proxy gateway along with its own OpenAPI spec for those still needing them to consume your APIs by using the appropriate plugin.

There are some disadvantages to using gRPC:

  • Being a relatively recent technology it has some complexities in learning, setting up and use. This may especially be true for developers coming from more traditional technologies (like REST)
  • Browser support for gRPC may be limited. Even though web clients can be generated opening up access to non custom ports (hosting gRPC services) may not be feasible due to org security policies.

Despite this (we feel) its advantages outweight the disadvantages. Improved tooling over time, increase in familiriaty and a robust plugin ecosystem all have made gRPC a popular choice. The browser support limitation will be addressed in the next article in this series.

In this article we will build a simple gRPC service to show case common features and patterns. This will server as a foundation for the upcoming guides in this series. Let us get started!

Motivating Example

Let us build a simple service for power a group chat application (eg like Whatsapp or Zulip or Slack or Teams etc). Our goal is not to displace any of the existing popular services but rather to demonstrate the various aspects in a robust service powering a popular application genre. Our chat service - OneHub - is simple enough. It has:

  • Topics - A place where a group of related users (by team, project, or interest) can share messages to communicate with each other. Very similar to (but also much more simpler than) channels in Slack or Microsoft.
  • Messages - The message being sent in the topic by users.

(Kudos if you have noticed that the "User" is missing. For now we will ignore logins/auth and treat users simply an opaque user id. This will simply testing our service across a number of features without worrying about login mechanisms etc. We will come to all things about Auth, User management and even social features in a future article). This service is rudimentary yet provides enough scope to take it in several directions which will be the topic of future dedicated posts.

Prerequisites

This tutorial assumes you have the following already installed:

  • golang (1.18+)
  • Install protoc. On OSX it is as simple as brew install protobuf
  • gRPC protoc tools for generating Go
    • go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
    • go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Optional - We wont go too much into building services outside Go but just for fun we will also generate some of the Python stubs to show easy it all is and if there is popular demand one day there could be an extension to this series covering other languages in more detail.

  • gRPC protoc tools for generating Python
    • pyenv virtualenv onehub
    • pyenv activate onehub
    • pip install grpcio
    • pip install grpcio-tools

Setup your project

Code for this can already be found in the OneHub repo. The repo is organized by service and branches are used as checkpoints aligned with end of each part in this series for easier revisiting.

mkdir onehub
cd onehub
# Replace this with your own github repo path
go mod init github.com/panyam/onehub
mkdir -p protos/onehub/v1
touch protos/onehub/v1/models.proto
touch protos/onehub/v1/messages.proto
touch protos/onehub/v1/topics.proto

# Install dependencies
go get google.golang.org/grpc
go get github.com/stretchr/testify/assert

Note when creating your protos it is good practise to have them versioned (v1 above).

There are several ways to organize your protos, for example (and not limited to):

  1. One giant proto for the entire service encompassing all models and protos (eg onehub.proto)
  2. All Foo entity related models and services in foo.proto
  3. All models in a single proto (models.proto) accompanied by services for entity Foo in foo.proto.

In this series we are using the 3rd approach as it also allows us to share models across service while still seperating the individual entity services cleanly.

Define your Service

The .proto files are the starting point of a gRPC, so we can start there with some basic details:

Models

syntax = "proto3";
import "google/protobuf/timestamp.proto";
import "google/protobuf/struct.proto";

option go_package = "github.com/onehub/protos";
package onehub.v1;

// Artists perform/play/sing songs
message Topic {
  google.protobuf.Timestamp created_at = 1;
  google.protobuf.Timestamp updated_at = 2;

  // ID of the topic
  string id = 3;

  // ID of the user that created this topic
  string creator_id = 4;
  
  // A unique name of the topic that users can use to connect to
  string name = 5;

  // IDs of users in this topic.   Right now no information about
  // their participation is kept.
  repeated string users = 6;
}

/**
 * An individual message in a topic
 */
message Message {
  /**
   * When the message was created on the server.
   */
  google.protobuf.Timestamp created_at = 1;

  /**
   * When the message or its body were last modified (if modifications are
   * possible).
   */
  google.protobuf.Timestamp updated_at = 2;

  /**
   * ID of the message guaranteed to be unique within a topic.
   * Set only by the server and cannot be modified.
   */
  string id = 3;

  /**
   * User sending this message.
   */
  string user_id = 4;

  /**
   * Topic the message is part of.  This is only set by the server
   * and cannot be modified.
   */
  string topic_id = 5;

  /**
   * Content type of the message. Can be like a ContentType http
   * header or something custom like shell/command
   */
  string content_type = 6;

  /**
   * A simple way to just send text.
   */
  string content_text = 7;

  // Raw contents for data stored locally as JSON
  // Note we can have a combination of text, url and data
  // to show different things in the View/UI
  google.protobuf.Struct content_data = 8;
}

Topic Service

syntax = "proto3";
import "google/protobuf/field_mask.proto";

option go_package = "github.com/onehub/protos";
package onehub.v1;

import "onehub/v1/models.proto";

/**
 * Service for operating on topics
 */
service TopicService {
  /**
   * Create a new sesssion
   */
  rpc CreateTopic(CreateTopicRequest) returns (CreateTopicResponse) {
  }

  /**
   * List all topics from a user.
   */
  rpc ListTopics(ListTopicsRequest) returns (ListTopicsResponse) { 
  }

  /**
   * Get a particular topic
   */
  rpc GetTopic(GetTopicRequest) returns (GetTopicResponse) { 
  }

  /**
   * Batch get multiple topics by ID
   */
  rpc GetTopics(GetTopicsRequest) returns (GetTopicsResponse) { 
  }

  /**
   * Delete a particular topic
   */
  rpc DeleteTopic(DeleteTopicRequest) returns (DeleteTopicResponse) { 
  }

  /**
   * Updates specific fields of a topic
   */
  rpc UpdateTopic(UpdateTopicRequest) returns (UpdateTopicResponse) {
  }
}

/**
 * Topic creation request object
 */
message CreateTopicRequest {
  /**
   * Topic being updated
   */
  Topic topic = 1;
}

/**
 * Response of an topic creation.
 */
message CreateTopicResponse {
  /**
   * Topic being created
   */
  Topic topic = 1;
}

/**
 * An topic search request.  For now only paginations params are provided.
 */
message ListTopicsRequest {
  /**
   * Instead of an offset an abstract  "page" key is provided that offers
   * an opaque "pointer" into some offset in a result set.
   */
  string page_key = 1;

  /**
   * Number of results to return.
   */
  int32 page_size = 2;
}

/**
 * Response of a topic search/listing.
 */
message ListTopicsResponse {
  /**
   * The list of topics found as part of this response.
   */
  repeated Topic topics = 1;

  /**
   * The key/pointer string that subsequent List requests should pass to
   * continue the pagination.
   */
  string next_page_key = 2;
}

/**
 * Request to get an topic.
 */
message GetTopicRequest {
  /**
   * ID of the topic to be fetched
   */
  string id = 1;
}

/**
 * Topic get response
 */
message GetTopicResponse {
  Topic topic = 1;
}

/**
 * Request to batch get topics
 */
message GetTopicsRequest {
  /**
   * IDs of the topic to be fetched
   */
  repeated string ids = 1;
}

/**
 * Topic batch-get response
 */
message GetTopicsResponse {
  map<string, Topic> topics = 1;
}

/**
 * Request to delete an topic.
 */
message DeleteTopicRequest {
  /**
   * ID of the topic to be deleted.
   */
  string id = 1;
}

/**
 * Topic deletion response
 */
message DeleteTopicResponse {
}

/**
 * The request for (partially) updating an Topic.
 */
message UpdateTopicRequest {
  /**
   * Topic being updated
   */
  Topic topic = 1;

  /**
   * Mask of fields being updated in this Topic to make partial changes.
   */
  google.protobuf.FieldMask update_mask = 2;

  /**
   * IDs of users to be added to this topic.
   */
  repeated string add_users = 3;

  /**
   * IDs of users to be removed from this topic.
   */
  repeated string remove_users = 4;
}

/**
 * The request for (partially) updating an Topic.
 */
message UpdateTopicResponse {
  /**
   * Topic being updated
   */
  Topic topic = 1;
}

Message Service

syntax = "proto3";
import "google/protobuf/field_mask.proto";

option go_package = "github.com/onehub/protos";
package onehub.v1;

import "onehub/v1/models.proto";

/**
 * Service for operating on messages
 */
service MessageService {
  /**
   * Create a new sesssion
   */
  rpc CreateMessage(CreateMessageRequest) returns (CreateMessageResponse) {
  }

  /**
   * List all messages in a topic
   */
  rpc ListMessages(ListMessagesRequest) returns (ListMessagesResponse) { 
  }

  /**
   * Get a particular message
   */
  rpc GetMessage(GetMessageRequest) returns (GetMessageResponse) { 
  }

  /**
   * Batch get multiple messages by IDs
   */
  rpc GetMessages(GetMessagesRequest) returns (GetMessagesResponse) { 
  }

  /**
   * Delete a particular message
   */
  rpc DeleteMessage(DeleteMessageRequest) returns (DeleteMessageResponse) { 
  }

  /**
   * Update a message within a topic.
   */
  rpc UpdateMessage(UpdateMessageRequest) returns (UpdateMessageResponse) {
  }
}

/**
 * Message creation request object
 */
message CreateMessageRequest {
  /**
   * Message being updated
   */
  Message message = 1;
}

/**
 * Response of an message creation.
 */
message CreateMessageResponse {
  /**
   * Message being created
   */
  Message message = 1;
}

/**
 * A message listing request.  For now only paginations params are provided.
 */
message ListMessagesRequest {
  /**
   * Instead of an offset an abstract  "page" key is provided that offers
   * an opaque "pointer" into some offset in a result set.
   */
  string page_key = 1;

  /**
   * Number of results to return.
   */
  int32 page_size = 2;

  /**
   * Topic in which messages are to be listed.  Required.
   */
  string topic_id = 3;
}

/**
 * Response of a topic search/listing.
 */
message ListMessagesResponse {
  /**
   * The list of topics found as part of this response.
   */
  repeated Message messages = 1;

  /**
   * The key/pointer string that subsequent List requests should pass to
   * continue the pagination.
   */
  string next_page_key = 2;
}

/**
 * Request to get a single message.
 */
message GetMessageRequest {
  /**
   * ID of the topic to be fetched
   */
  string id = 1;
}

/**
 * Message get response
 */
message GetMessageResponse {
  Message message = 1;
}

/**
 * Request to batch get messages
 */
message GetMessagesRequest {
  /**
   * IDs of the messages to be fetched
   */
  repeated string ids = 1;
}

/**
 * Message batch-get response
 */
message GetMessagesResponse {
  map<string, Message> messages = 1;
}

/**
 * Request to delete an message
 */
message DeleteMessageRequest {
  /**
   * ID of the message to be deleted.
   */
  string id = 1;
}

/**
 * Message deletion response
 */
message DeleteMessageResponse {
}

message UpdateMessageRequest {
  // The message being updated.  The topic ID AND message ID fields *must*
  // be specified in this message object.  How other fields are used is
  // determined by the update_mask parameter enabling partial updates
  Message message = 1;

  // Indicates which fields are being updated
  // If the field_mask is *not* provided then we reject
  // a replace (as required by the standard convention) to prevent
  // full replace in error.  Instead an update_mask of "*" must be passed.
  google.protobuf.FieldMask update_mask = 3;

  // Any fields specified here will be "appended" to instead of being
  // replaced
  google.protobuf.FieldMask append_mask = 4;
}

message UpdateMessageResponse {
  // The updated message
  Message message = 1;
}

Note each entity was relegated to its own service - though this does not translate to a seperate server (or even process). This is merely for convinience.

For the most part, resource oriented designs have been adopted for the entities, their respective services and methods. As a summary:

  • Entities (in models.proto) have an id field to denote their primary key/object ID
  • All entities have a created/updated timestamp which are set in the create and update methods respectively.
  • All services have the typical CRUD methods.
  • The methods (rpcs) in each service follow similar patterns for their CRUD methods, eg:
    • FooService.Get => method(GetFooRequest) => GetFooResponse
    • FooService.Delete => method(DeleteFooRequest) => DeleteFooResponse
    • FooService.Create => method(CreateFooRequest) => CreateFooResponse
    • FooService.Update => method(UpdateFooRequest) => UpdateFooResponse
  • FooServer.Create methods take a Foo instance and set the instances id, created_at and updated_at fields
  • FooService.Update methods take a Foo instance along with a update_mask to highlight fields being changed and updates the fields. Additionally it also ignores the id method so an id cannot be over-written.

The entities (and relationships) are very straightforward. Some (slightly) noteworthy aspects are:

  1. Topics have a list of ids representing the participants in the topic (we are not focussing on the scalability bottlenecks from a large number of users in a Topic yet).
  2. Messages hold a reference to the Topic (via topic_id).
  3. The Message is very simple and only supports text messages (along with a way to pass in extra information or slightly custom message types - via content_data).

Generate the service stubs and clients

The protoc command-line tool ensures that server (stubs) and clients are generated from this basic stub.

The magic of protoc is that it does not generate anything on its own. Instead it uses plugins for different "purposes" to generate custom artifacts. First let us generate go artifacts:

SRC_DIR=<ABSOLUTE_PATH_OF_ONEHUB>
PROTO_DIR:=$SRC_DIR/protos
OUT_DIR:=$SRC_DIR/gen/go
protoc --go_out=$OUT_DIR --go_opt=paths=source_relative               \
       --go-grpc_out=$OUT_DIR --go-grpc_opt=paths=source_relative     \
       --proto_path=$(PROTO_DIR)                                      \
        $PROTO_DIR/onehub/v1/*.proto

This is quite cumbersome so we can add this into a Makefile and simply run make all to generate protos and build everything going forward.

Makefile


# Some vars to detemrine go locations etc
GOROOT=$(which go)
GOPATH=$(HOME)/go
GOBIN=$(GOPATH)/bin

# Evaluates the abs path of the directory where this Makefile resides
SRC_DIR:=$(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))

# Where the protos exist
PROTO_DIR:=$(SRC_DIR)/protos

# where we want to generate server stubs, clients etc
OUT_DIR:=$(SRC_DIR)/gen/go

all: printenv goprotos

goprotos:
	echo "Generating GO bindings"
	rm -Rf $(OUT_DIR) && mkdir -p $(OUT_DIR)
	protoc --go_out=$(OUT_DIR) --go_opt=paths=source_relative          	\
       --go-grpc_out=$(OUT_DIR) --go-grpc_opt=paths=source_relative		\
       --proto_path=$(PROTO_DIR) 																			\
      $(PROTO_DIR)/onehub/v1/*.proto

printenv:
	@echo MAKEFILE_LIST=$(MAKEFILE_LIST)
	@echo SRC_DIR=$(SRC_DIR)
	@echo PROTO_DIR=$(PROTO_DIR)
	@echo OUT_DIR=$(OUT_DIR)
	@echo GOROOT=$(GOROOT)
	@echo GOPATH=$(GOPATH)
	@echo GOBIN=$(GOBIN)

Now all generated stubs can be found in the gen/go/onehub/v1 folder (because our protos folder hosted the service defs within onehub/v1).

Briefly the following are created:

  • For every X.proto file a gen/go/onehub/v1/X.pb.go is created. This file contains the model definition of every "message" in the .proto file (eg Topic and Message).
  • For every Y.proto file that contains service definitions a X_grpc.pb.go file is generated that contains:
    • A server interface that must be implemented (coming in the next section).
    • For a service X, an interface called XService is generated where the methods are all the rpc methods stipulated in the Y.proto file.
    • A client is generated that can talk to a running implementation of the XService interface (coming below).

Pretty powerful isn't it! Now let us look at actually implementing the services.

Implementing your Service

Our services are very simple. They store the different instances in memory as a simple collection of elements added in the order in which they were created (we will look at using a real database in the next part) and serve them by querying and updating this collection. Since all the services have (mostly) similar implementations (CRUD) a base store object has been created to represent the in-memory collection and the services simply use this store.

This (simple) base entity looks like:

Base Entity Store

package services

import (
	"fmt"
	"log"
	"sort"
	"time"

	tspb "google.golang.org/protobuf/types/known/timestamppb"
)

type EntityStore[T any] struct {
	IDCount  int
	Entities map[string]*T

	// Getters/Setters for ID
	IDSetter func(entity *T, id string)
	IDGetter func(entity *T) string

	// Getters/Setters for created timestamp
	CreatedAtSetter func(entity *T, ts *tspb.Timestamp)
	CreatedAtGetter func(entity *T) *tspb.Timestamp

	// Getters/Setters for udpated timestamp
	UpdatedAtSetter func(entity *T, ts *tspb.Timestamp)
	UpdatedAtGetter func(entity *T) *tspb.Timestamp
}

func NewEntityStore[T any]() *EntityStore[T] {
	return &EntityStore[T]{
		Entities: make(map[string]*T),
	}
}

func (s *EntityStore[T]) Create(entity *T) *T {
	s.IDCount++
	newid := fmt.Sprintf("%d", s.IDCount)
	s.Entities[newid] = entity
	s.IDSetter(entity, newid)
	s.CreatedAtSetter(entity, tspb.New(time.Now()))
	s.UpdatedAtSetter(entity, tspb.New(time.Now()))
	return entity
}

func (s *EntityStore[T]) Get(id string) *T {
	if entity, ok := s.Entities[id]; ok {
		return entity
	}
	return nil
}

func (s *EntityStore[T]) BatchGet(ids []string) map[string]*T {
	out := make(map[string]*T)
	for _, id := range ids {
		if entity, ok := s.Entities[id]; ok {
			out[id] = entity
		}
	}
	return out
}

// Updates specific fields of an Entity
func (s *EntityStore[T]) Update(entity *T) *T {
	s.UpdatedAtSetter(entity, tspb.New(time.Now()))
	return entity
}

// Deletes an entity from our system.
func (s *EntityStore[T]) Delete(id string) bool {
	_, ok := s.Entities[id]
	if ok {
		delete(s.Entities, id)
	}
	return ok
}

// Finds and retrieves entity matching the particular criteria.
func (s *EntityStore[T]) List(ltfunc func(t1, t2 *T) bool, filterfunc func(t *T) bool) (out []*T) {
	log.Println("E: ", s.Entities)
	for _, ent := range s.Entities {
		if filterfunc == nil || filterfunc(ent) {
			out = append(out, ent)
		}
	}
	// Sort in reverse order of name
	sort.Slice(out, func(idx1, idx2 int) bool {
		ent1 := out[idx1]
		ent2 := out[idx2]
		return ltfunc(ent1, ent2)
	})
	return
}

Using this the Topic service is now very simple:

Topic Service Implementation

package services

import (
	"context"
	"log"
	"strings"

	protos "github.com/panyam/onehub/gen/go/onehub/v1"
	tspb "google.golang.org/protobuf/types/known/timestamppb"
)

type TopicService struct {
	protos.UnimplementedTopicServiceServer
	*EntityStore[protos.Topic]
}

func NewTopicService(estore *EntityStore[protos.Topic]) *TopicService {
	if estore == nil {
		estore = NewEntityStore[protos.Topic]()
	}
	estore.IDSetter = func(topic *protos.Topic, id string) { topic.Id = id }
	estore.IDGetter = func(topic *protos.Topic) string { return topic.Id }

	estore.CreatedAtSetter = func(topic *protos.Topic, val *tspb.Timestamp) { topic.CreatedAt = val }
	estore.CreatedAtGetter = func(topic *protos.Topic) *tspb.Timestamp { return topic.CreatedAt }

	estore.UpdatedAtSetter = func(topic *protos.Topic, val *tspb.Timestamp) { topic.UpdatedAt = val }
	estore.UpdatedAtGetter = func(topic *protos.Topic) *tspb.Timestamp { return topic.UpdatedAt }

	return &TopicService{
		EntityStore: estore,
	}
}

// Create a new Topic
func (s *TopicService) CreateTopic(ctx context.Context, req *protos.CreateTopicRequest) (resp *protos.CreateTopicResponse, err error) {
	resp = &protos.CreateTopicResponse{}
	resp.Topic = s.EntityStore.Create(req.Topic)
	return
}

// Get a single topic by id
func (s *TopicService) GetTopic(ctx context.Context, req *protos.GetTopicRequest) (resp *protos.GetTopicResponse, err error) {
	log.Println("Getting Topic by ID: ", req.Id)
	resp = &protos.GetTopicResponse{
		Topic: s.EntityStore.Get(req.Id),
	}
	return
}

// Batch gets multiple topics.
func (s *TopicService) GetTopics(ctx context.Context, req *protos.GetTopicsRequest) (resp *protos.GetTopicsResponse, err error) {
	log.Println("BatchGet for IDs: ", req.Ids)
	resp = &protos.GetTopicsResponse{
		Topics: s.EntityStore.BatchGet(req.Ids),
	}
	return
}

// Updates specific fields of an Topic
func (s *TopicService) UpdateTopic(ctx context.Context, req *protos.UpdateTopicRequest) (resp *protos.UpdateTopicResponse, err error) {
	resp = &protos.UpdateTopicResponse{
		Topic: s.EntityStore.Update(req.Topic),
	}
	return
}

// Deletes an topic from our system.
func (s *TopicService) DeleteTopic(ctx context.Context, req *protos.DeleteTopicRequest) (resp *protos.DeleteTopicResponse, err error) {
	resp = &protos.DeleteTopicResponse{}
	s.EntityStore.Delete(req.Id)
	return
}

// Finds and retrieves topics matching the particular criteria.
func (s *TopicService) ListTopics(ctx context.Context, req *protos.ListTopicsRequest) (resp *protos.ListTopicsResponse, err error) {
	results := s.EntityStore.List(func(s1, s2 *protos.Topic) bool {
		return strings.Compare(s1.Name, s2.Name) < 0
	}, nil)
	log.Println("Found Topics: ", results)
	resp = &protos.ListTopicsResponse{Topics: results}
	return
}

The Message service is also eerily similar and can here.

Wrap it all with a runner

We have implemented the services with our logic but the services need to be brought up.

The general steps are:

  • create a GRPC Server instance
  • register each our service implementations with this server
  • run this server on a specific port

Main Server CLI

package main

import (
	"flag"
	"log"
	"net"

	"google.golang.org/grpc"

	v1 "github.com/panyam/onehub/gen/go/onehub/v1"
	svc "github.com/panyam/onehub/services"

	// This is needed to enable the use of the grpc_cli tool
	"google.golang.org/grpc/reflection"
)

var (
	addr = flag.String("addr", ":9000", "Address to start the onehub grpc server on.")
)

func startGRPCServer(addr string) {
	// create new gRPC server
	server := grpc.NewServer()
	v1.RegisterTopicServiceServer(server, svc.NewTopicService(nil))
	v1.RegisterMessageServiceServer(server, svc.NewMessageService(nil))
	if l, err := net.Listen("tcp", addr); err != nil {
		log.Fatalf("error in listening on port %s: %v", addr, err)
	} else {
		// the gRPC server
		log.Printf("Starting grpc endpoint on %s:", addr)
		reflection.Register(server)
		if err := server.Serve(l); err != nil {
			log.Fatal("unable to start server", err)
		}
	}
}

func main() {
	flag.Parse()
	startGRPCServer(*addr)
}

This server can now be run (by default on port 9000) with:

go run cmd/server.go

Note this is a simple service with Unary RPC methods. ie the client sends a single request to the server and waits for a single response. There are also other types of methods

  1. Server Streaming RPC - The client sends a request to the server and receives a stream of responses (similar to long-polling in http where the client listens to chunks of messages on the open connection).
  2. Client streaming RPC - Here the client sends a stream of messages in a single request and receives a single response from the server. For example a single request from the client could involve multiple location updates (spread out over time) and the response from the server could be a single "path" object the client travelled along.
  3. Bidirectional streaming RPC - The client initiates a connection with the server and both the client and server can send messages independent of one another. The similarity for this in the HTTP universe would be Websocket connections.

We will implement one or more of these in future tutorials.

Client calls to the server

Now it is time to test our server. Note the grpc server is not a REST endpoint. So curl would not work (we will cover this in Part 2). We can make calls against the server in a couple of ways - using a CLI utility (much like curl for REST/http services) or by using the clients generated by the protoc tool. Even better, we can also make client calls from other languages - if we had opted to generate libraries targeting those languages too.

Calling the server via grpc_cli utility

A grpc client (grpc_cli) exists to make direct calls from the command line. On OSX, this can be installed with brew install grpc.

If the server is not running then go ahead and start it (as per the previous section). We can now start calling operations on the server itself - either to make calls or reflect on it!

List all operations

grpc_cli ls localhost:9000 -l

filename: grpc/reflection/v1/reflection.proto
package: grpc.reflection.v1;
service ServerReflection {
  rpc ServerReflectionInfo(stream grpc.reflection.v1.ServerReflectionRequest) returns (stream grpc.reflection.v1.ServerReflectionResponse) {}
}

filename: grpc/reflection/v1alpha/reflection.proto
package: grpc.reflection.v1alpha;
service ServerReflection {
  rpc ServerReflectionInfo(stream grpc.reflection.v1alpha.ServerReflectionRequest) returns (stream grpc.reflection.v1alpha.ServerReflectionResponse) {}
}

filename: onehub/v1/messages.proto
package: onehub.v1;
service MessageService {
  rpc CreateMessage(onehub.v1.CreateMessageRequest) returns (onehub.v1.CreateMessageResponse) {}
  rpc ListMessages(onehub.v1.ListMessagesRequest) returns (onehub.v1.ListMessagesResponse) {}
  rpc GetMessage(onehub.v1.GetMessageRequest) returns (onehub.v1.GetMessageResponse) {}
  rpc GetMessages(onehub.v1.GetMessagesRequest) returns (onehub.v1.GetMessagesResponse) {}
  rpc DeleteMessage(onehub.v1.DeleteMessageRequest) returns (onehub.v1.DeleteMessageResponse) {}
  rpc UpdateMessage(onehub.v1.UpdateMessageRequest) returns (onehub.v1.UpdateMessageResponse) {}
}

filename: onehub/v1/topics.proto
package: onehub.v1;
service TopicService {
  rpc CreateTopic(onehub.v1.CreateTopicRequest) returns (onehub.v1.CreateTopicResponse) {}
  rpc ListTopics(onehub.v1.ListTopicsRequest) returns (onehub.v1.ListTopicsResponse) {}
  rpc GetTopic(onehub.v1.GetTopicRequest) returns (onehub.v1.GetTopicResponse) {}
  rpc GetTopics(onehub.v1.GetTopicsRequest) returns (onehub.v1.GetTopicsResponse) {}
  rpc DeleteTopic(onehub.v1.DeleteTopicRequest) returns (onehub.v1.DeleteTopicResponse) {}
  rpc UpdateTopic(onehub.v1.UpdateTopicRequest) returns (onehub.v1.UpdateTopicResponse) {}
}

Create a Topic

grpc_cli --json_input --json_output call localhost:9000 CreateTopic '{topic: {name: "First Topic", creator_id: "user1"}}'

{
 "topic": {
  "createdAt": "2023-07-28T07:30:54.633005Z",
  "updatedAt": "2023-07-28T07:30:54.633006Z",
  "id": "1",
  "creatorId": "user1",
  "name": "First Topic"
 }
}

and another

grpc_cli --json_input --json_output call localhost:9000 CreateTopic '{topic: {name: "Urgent topic", creator_id: "user2", users: ["user1", "user2", "user3"]}}'

{
 "topic": {
  "createdAt": "2023-07-28T07:32:04.821800Z",
  "updatedAt": "2023-07-28T07:32:04.821801Z",
  "id": "2",
  "creatorId": "user2",
  "name": "Urgent topic",
  "users": [
   "user1",
   "user2",
   "user3"
  ]
 }
}

List all topics

grpc_cli --json_input --json_output call localhost:9000 ListTopics {}

{
 "topics": [
  {
   "createdAt": "2023-07-28T07:30:54.633005Z",
   "updatedAt": "2023-07-28T07:30:54.633006Z",
   "id": "1",
   "creatorId": "user1",
   "name": "First Topic"
  },
  {
   "createdAt": "2023-07-28T07:32:04.821800Z",
   "updatedAt": "2023-07-28T07:32:04.821801Z",
   "id": "2",
   "creatorId": "user2",
   "name": "Urgent topic",
   "users": [
    "user1",
    "user2",
    "user3"
   ]
  }
 ]
}

Get Topics by IDs

grpc_cli --json_input --json_output call localhost:9000 GetTopics '{"ids": ["1", "2"]}'

{
 "topics": {
  "1": {
   "createdAt": "2023-07-28T07:30:54.633005Z",
   "updatedAt": "2023-07-28T07:30:54.633006Z",
   "id": "1",
   "creatorId": "user1",
   "name": "First Topic"
  },
  "2": {
   "createdAt": "2023-07-28T07:32:04.821800Z",
   "updatedAt": "2023-07-28T07:32:04.821801Z",
   "id": "2",
   "creatorId": "user2",
   "name": "Urgent topic",
   "users": [
    "user1",
    "user2",
    "user3"
   ]
  }
 }
}

Delete a topic followed by a Listing

grpc_cli --json_input --json_output call localhost:9000 DeleteTopic '{"id": "1"}'

connecting to localhost:9000
{}
Rpc succeeded with OK status

grpc_cli --json_input --json_output call localhost:9000 ListTopics {}

{
 "topics": [
  {
   "createdAt": "2023-07-28T07:32:04.821800Z",
   "updatedAt": "2023-07-28T07:32:04.821801Z",
   "id": "2",
   "creatorId": "user2",
   "name": "Urgent topic",
   "users": [
    "user1",
    "user2",
    "user3"
   ]
  }
 ]
}

Programmatically calling the server

Instead of going into this deep, the tests in the service folder show how clients can be created as well as in how to write tests.

Conclusion

That was a lot to cover but we made it. Even though it was a basic example, (hopefully) it set a good foundation for the topics in the rest of the series.

In summary, gRPC is a crucial component of modern software development that allows developers to build high-performance, scalable, and interoperable systems. Its features and benefits have made it a popular choice among companies that need to handle large amounts of data or need to support real-time communication across multiple platforms.

  • Created a grpc service in Go from the ground up, with a very simple implementation (sadly lacking in persistence),
  • Exercised the CLI utility to make calls to the running service
  • Exercised the generated clients while also writing tests

In the next article we will explore how to generate REST/Http bindings for a grpc service. This would enable us to use standard http clients (like curl) to access our service.