The Makefile

Some people might be turned off by the use of a makefile. However, it is going to make this significantly more convenient, especially as your project grows.

GOC=go
GO111MODULE=on

DIRECTORIES=$(sort $(dir $(wildcard pkg/*/*/)))
MOCKS=$(foreach x, $(DIRECTORIES), mocks/$(x))

.PHONY: all test clean-mocks mocks

all: demo

demo:
	$(GOC) build -o demo

test: | mocks
	go test ./...

clean-mocks:
	rm -rf mocks

mocks: $(MOCKS)
	
$(MOCKS): mocks/% : %
	mockery -output=$@ -dir=$^ -all

This makefile takes all of the subdirectories which match the pattern “pkg/*/*” and then matches them with an output to be in the folder mocks/pkg/... . For each of these pairs, it makes a call to mockery with the -all flag. This flag tells it to mock all of the interfaces in a directory.

The Go Code to Mock

For this example, we will have a Teller struct, which exports a method Tell. This method takes in a story and reads it chapter by chapter to an audience far and wide. Here is what the story interface and Teller struct look like.

type Story interface {
	ReadChapter(chapter int) ([]byte, error)
	HasChapter(chapter int) bool
}

type Teller struct {
}

func (teller Teller) say(words []byte) {
	//Pretend this makes it talk
}

func (teller Teller) Tell(story Story) error {
	for i := 0; story.HasChapter(i); i++ {
		words, err := story.ReadChapter(i)
		if err != nil {
			return err
		}
		teller.say(words)
	}
	return nil
}

How can you make sure that the Teller reads every single chapter until there are no more in the story? You mock the story! First, you must generate the mocks with the command make mocks (or you can just run the test with make test, it knows to make the calls to mockery first). Here is our tester code for the Teller.

import (
	"testing"

	demoMock "github.com/moduledemos/mockery-demo/mocks/pkg/demo"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/mock"
)

func TestTeller_Tell(t *testing.T) {
	teller := Teller{}
	story := new(demoMock.Story)
	noChapters := 20
	story.On("ReadChapter", mock.Anything).Return(
		[]byte(":)"), nil).Times(noChapters)
	for i := 0; i < noChapters; i++ {
		story.On("HasChapter", i).Return(true).Once()
	}
	story.On("HasChapter", noChapters).Return(false).Once()
	err := teller.Tell(story)
	assert.NoError(t, err)

	story.AssertExpectations(t)
}

You can see that we can mock different return values for the HasChapter function based on the input given to it. Additionally, you might notice .Times(n) or .Once appended to the call to .On. When combined with the call to AssertExpectations at the end, this enforces the number of times each variation is called.

If it does not matter what the input is, you can use the global mock.Anything from the testify library to match with any given input. With all of these features, I was able to create a test that ensured that all of the chapters would be read until the function HasChapter returned false. All of this, in just ~12 lines of code. If you like doing more work with less code, then I highly recommend using mockery and testify for all of your Go testing needs.

The post Mockery; Mock Everything In Go. appeared first on Module Safari.

Assert

Testify’s assert package contains functions that make checking certain conditions easy. They replace the normal if statement list flow and have reflection built-in for comparisons between structs. For this example, let’s say we are making a function fib, which calculates the number in the nth place in the Fibonacci Sequence. Here is our naive implementation of this, which has a few issues.

func fib(n int64) int64 {
	var a int64 = 0
	var b int64 = 1
	for i := int64(0); i < n-1; i++ {
		tmp := b
		b = a + b
		a = tmp
	}
	return b
}

We want to be sure that this implementation is correct, so we add some tests for it using testify.

package main

import(
	"testing"

	"github.com/stretchr/testify/assert"
)

func TestFib(t *testing.T) {
	assert.Equal(t, int64(0), fib(0))
	assert.Equal(t, int64(1), fib(1))
	assert.Equal(t, int64(1), fib(2))
	assert.Equal(t, int64(6765), fib(20))
}

Now, we run our tests with go test and we can immediately see an issue.

--- FAIL: TestFib (0.00s)
    fib_test.go:12: 
        	Error Trace:	fib_test.go:10
        	Error:      	Not equal: 
        	            	expected: 0
        	            	actual  : 1
        	Test:       	TestFib
FAIL

Thanks to testify, we easily see that on line 10, the output of the function was 1, instead of 0. We go to our code and see that we don’t handle the case of n == 0, and fix the issue.

func fib(n int64) int64 {
	var a int64 = 0
	var b int64 = 1
	for i := int64(0); i < n; i++ {
		tmp := b
		b = a + b
		a = tmp
	}
	return a
}

The tests passed, and we can be more confident that our code is correct.

Require

While assert is great for most cases, sometimes you have dependencies in your test function. There, you want to make sure that when a test fails, it reports only what failed. Enter the require package. Require feels like a carbon copy of the assert package, with just one key difference: it calls t.FailNow(). Let’s make a function which uses the Fibonacci function from earlier, and outputs all terms up to n in an array.

func fibAll(n int64) []int64 {
	if n > 0 {
		return nil
	}
	out := make([]int64, n + 1)
	for i := int64(0); i <= n; i++ {
		out[i] = fib(i)
	}
	return out
}

This function needs some tests, so let’s write some up real quick. We want to do some spot checking to make sure that the sequence is generated correctly.

func TestFibAll(t *testing.T) {
	for i := int64(15); i < 40; i++ {
		fibNumbers := fibAll(i)
		assert.NotNil(t, fibNumbers)
		assert.Len(t, fibNumbers, int(i+1))
		assert.Equal(t, 233, fibNumbers[13])
		assert.Equal(t, 55, fibNumbers[10])
	}
}

We run our tests and get back this messy error.

--- FAIL: TestFibAll (0.00s)
    fib_test.go:19: 
        	Error Trace:	fib_test.go:19
        	Error:      	Expected value not to be nil.
        	Test:       	TestFibAll
    fib_test.go:20: 
        	Error Trace:	fib_test.go:20
        	Error:      	"[]" should have 16 item(s), but has 0
        	Test:       	TestFibAll
panic: runtime error: index out of range [13] with length 0 [recovered]
	panic: runtime error: index out of range [13] with length 0

goroutine 21 [running]:
testing.tRunner.func1(0xc000116200)
	/snap/go/4762/src/testing/testing.go:874 +0x3a3
panic(0x719700, 0xc0000d42a0)
[Full stack trace omitted]
exit status 2
FAIL

As you can see, the tests reported 2 failures instead of just one, and the code panicked. Of course, this is still usable and would be significantly better than nothing. However, if the unit tests were more complex, it can start to become unclear as to what the actual problem is. So, let’s put “require” to use.

func TestFibAll(t *testing.T) {
	for i := int64(15); i < 40; i++ {
		fibNumbers := fibAll(i)
		require.NotNil(t, fibNumbers)
		require.Len(t, fibNumbers, int(i+1))
		assert.Equal(t, 233, fibNumbers[13])
		assert.Equal(t, 55, fibNumbers[10])
	}
}

When we run our tests, it becomes very clear what the issue is.

--- FAIL: TestFibAll (0.00s)
    fib_test.go:20: 
        	Error Trace:	fib_test.go:20
        	Error:      	Expected value not to be nil.
        	Test:       	TestFibAll
FAIL

This is a significant improvement. Especially as the complexity of a codebase increases, simplified test failures can save you a lot of time. The last thing anyone wants to do is dig through a ton of logs and code, just to find that the issue is an inverted comparison.

Overall, Testify is a great library for reducing the complexity of your Go test code. Every time I have used it, I have been extremely satisfied, as the functions do exactly what they say. I strongly suggest checking out the mock package and its use with mockery

The post Testify; Test Your Go Code appeared first on Module Safari.

Setup

I am going to glance over the setting up of the actual domain for this post, but the important part of this is that I have DNS set up correctly, and have a remote Linux host setup and running. Now, I just need to deploy the code to this host and see what happens, here is the code I used.

package main

import(
	"net/http"

	"github.com/gorilla/mux"
	"github.com/mholt/certmagic"
)

func main() {
	router := mux.NewRouter()
	router.HandleFunc("/", func(w http.ResponseWriter, r *http.Request){
		w.Write([]byte("Hello world"))
	}).Methods("GET", "OPTIONS")
	certmagic.HTTPS([]string{"certmagic.modulesafari.com"}, router)
}

As you can see, the Go code is extremely simple, the call to Certmagic that initiates all of the logic to get the cert is a very simple one-liner, and can handle multiple domains.

Execution

So, I put this code on the server, compiled it, and then ran it. Within 10 seconds, it had already completed the acme-challenge and obtained a certificate. That is very impressive! I was expected to have the time to get a cup of coffee. I barely had time to get out of my chair.

Of course, I had to check in the browser to make sure, and behold, it’s alive!

So, would I use Certmagic in production Golang services? It depends. I don’t imagine this being an effective solution if you are behind a load balancer. It also won’t work automagically if you are behind Cloudflare, I tried. The place were it really shines is when you have a very small service that you want to be secure. It is great for when you just want it to work on its own and not have to worry about it. It is definitely worth checking out, which you can do here. Props to mholt for such a cool library!

The post Certmagic; Automagic HTTPS for Golang appeared first on Module Safari.

Installing Boost using a package manager

The easiest way to install Boost is with your built-in package manager. It takes care of the installation of all of the dependencies and is more likely to be using a very stable version of the library. Also, there is the added benefit of it being easy to remove if you no longer want it.

Apt (Debian, Ubuntu, Pop OS, etc)

On most Debian based systems, you can install all of the Boost libraries with a single command.

apt-get install libboost-all-dev

DNF (Fedora, etc)

The newer versions of Fedora now use Dandified Yum, instead of just regular yum.

dnf install boost-devel

Yum (Centos, older versions of Fedora, etc)

yum install boost-devel

Manual Installation

If you require a different version of it than the one that is provided by your package manager, you can easily compile and install it manually.

Dependencies

Boost has very few dependencies. You mainly need just a C++ compiler and make.

• C++ compiler (g++, clang++, etc)
• make

Installation

With that out of the way, let’s get to the installation, do keep in mind that you will need to set the change the first few environment variables to what you would like them to be.

export MAJOR_VERSION=1
export MINOR_VERSION=70
export PATCH_VERSION=0
wget https://dl.bintray.com/boostorg/release/${MAJOR_VERSION}.${MINOR_VERSION}.${PATCH_VERSION}/source/boost_${MAJOR_VERSION}_${MINOR_VERSION}_${PATCH_VERSION}.tar.gz
tar -xzf boost_${MAJOR_VERSION}_${MINOR_VERSION}_${PATCH_VERSION}.tar.gz
cd boost_${MAJOR_VERSION}_${MINOR_VERSION}_${PATCH_VERSION}
./bootstrap.sh
./b2 install -j$(nproc)

You should now have successfully installed Boost on your machine. For more details, I suggest taking a look at the official boost documentation

The post How to install the Boost C++ libraries appeared first on Module Safari.

What is a Linux ulimit?

A ulimit or user limit in Linux is a restriction placed on a user’s access to certain resources. You can use the command ulimit -a to view them, this will give you a list of the ulimit values you can set, as well as their value. It is important to note that there are two types of limits, soft and hard.

Hard Limits

A hard limit is the maximum allowed to the user. It is set by the root user and acts as the hard ceiling for that value. The user will be unable to have a limit higher than this unless it is changed by the root user. There are a few ways to update this value, but the most common way is by updating the /etc/security/limits.conf file. In most situations, it is best to not update this value.

Soft Limits

A soft limit is the current maximum for the user. The user is able to change this at any time, either through setrlimit(2) or the ulimit command. Often times, you will only need to set this value, as very few situations call for resources above the hard limit.

Permanently Updating Limits

The best method for permanently updating soft limits and hard limits is via the /etc/security/limits.conf file. In this file, you can include the new soft limit defaults for a user, or update the hard limits. The format in which you insert your updates into this file is simple, it is just <domain> <type> <item> <value>. So, if you wanted to update the default file limit for the user apache, it would look like this.

apache soft nofile 5000

Now, all you would have to do is reboot the system, and the user apache’s file descriptor limit would default to 5000.

Temporarily Updating Limits

The easiest way to temporarily update a soft limit is with the ulimit command. If you wanted to set your file descriptor limit to 5000 for the duration of your shell session, you can use this command.

ulimit -S -n 5000

The -S flag signals that you are temporarily updating the soft limit value, and the -n flag states you wish to change the value for file-descriptors. You can also update hard limits using this method, just swap out the -S with -H. However, you can only set a lower value.

The post Learn how to set ulimit on Linux appeared first on Module Safari.

Getting To Hello World

Getting started is fairly standard for a from-source installation, you can also opt to compile it into a deb or rpm package via the build flags. Running the following commands installs served on your system. This installation requires Boost 1.53 or newer, if you do not have Boost installed on your system, you can install it using your favorite package manager or by following these instructions.

git clone https://github.com/meltwater/served.git
mkdir served/served.build && cd served/served.build
cmake ../served && make
sudo make install

Now, that you have it installed, it is time to build a simple webserver. This server will just give back “Hello world!” when you query against the endpoint GET /hello.

#include <served/served.hpp>

int main() {
	served::multiplexer mux;

	mux.handle("/hello")
		.get([](served::response & res, const served::request & req) {
			res << "Hello world!\n";
		});

	served::net::server server("0.0.0.0", "8080", mux);
	server.run(10);

	return 0;
}

To compile your program, you will need to link against the pthread, boost_system, and served shared objects, and use at least C++-11. On Linux, this would look roughly like the following.

g++ main.cpp -o demo -std=c++17 -pthread -lboost_system -lserved

Just run the binary and visit the URL http://localhost:8080/hello in your browser, and you have successfully reached the hello-world. All of the code for this demo is available on github.

Performance

Let’s take a quick look at the performance of C++ Served. At only 60k, the output binary from that demo is surprising small on my system. Of course, this is not the statically linked binary size. For this benchmark, we will be using Vegeta for load testing.

Requests      [total, rate, throughput]    99999, 20000.17, 19057.02
Duration      [total, attack, wait]        5.000256833s, 4.999908146s, 348.687µs
Latencies     [mean, 50, 90, 95, 99, max]  2.539834ms, 375.457µs, 7.240899ms, 8.184653ms, 38.421652ms, 1.691368283s
Bytes In      [total, mean]                1238770, 12.39
Bytes Out     [total, mean]                0, 0.00
Success       [ratio]                      95.29%
Status Codes  [code:count]                 0:4709  200:95290  

On my machine (Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 threads), the basic hello world server was able to handle around 19,000 requests per second! Additionally, it was able to maintain relatively low CPU usage during this attack, using only ~30% of the capacity available to it.

Requests      [total, rate, throughput]    150000, 30000.10, 14094.09
Duration      [total, attack, wait]        7.948649187s, 4.999982507s, 2.94866668s
Latencies     [mean, 50, 90, 95, 99, max]  9.756488ms, 5.137985ms, 7.529834ms, 7.989994ms, 51.273487ms, 6.782636245s
Bytes In      [total, mean]                1456377, 9.71
Bytes Out     [total, mean]                0, 0.00
Success       [ratio]                      74.69%
Status Codes  [code:count]                 0:37971  200:112029 

As the request rate increases above 20k, the performance started to degrade. Here, it seems like 20k requests per second is the sweet spot. Despite this, the performance of the library is very satisfactory under the given load. The majority of requests took under 1ms when it was under the 20k stress test. This should be more than enough to meet the needs of all but the most extreme demands.

The post Served; Build RESTful C++ servers appeared first on Module Safari.

package main

import(
	"github.com/sirupsen/logrus"
)

func main(){
	logrus.WithFields(logrus.Fields{
        	"foo":"bar"}).Info("hello world")
}

package main

import (
	log "github.com/sirupsen/logrus"
)

type msFormater struct{}

func (mf msFormater) Format(entry *log.Entry) ([]byte, error) {
	entry.Data["module"] = "safari"

	formatter := log.TextFormatter{}
	return formatter.Format(entry)
}

func main() {
	log.SetFormatter(msFormater{})
	log.SetLevel(log.DebugLevel)
	log.Info("hello")
}

BenchmarkLogrus-16                   	  358696      3222 ns/op
BenchmarkLog-16                      	 4048160       330 ns/op
BenchmarkLogrusMap-16                	  331330      3523 ns/op
BenchmarkLogMap-16                   	  638674      2005 ns/op
BenchmarkLogrusMapSmallMembers-16    	  326095      3625 ns/op
BenchmarkLogMapSmallMembers-16       	  624159      1935 ns/op
BenchmarkLogrusCallerReport-16       	  181448      6657 ns/op
BenchmarkLogCallerReport-16          	 1246078      1024 ns/op

BenchmarkLogrusBestCase-16           	    1518    759788 ns/op
BenchmarkLogBestCase-16              	    1750    683142 ns/op

The post Logrus; a structured logger for Go appeared first on Module Safari.