How should we prepare for Evil Inputs in Golang?

This is what owasp says: “The most common web application security weakness is the failure to properly validate input coming from the client or from the environment before using it. This weakness leads to almost all of the major vulnerabilities in web applications, such as cross site scripting, SQL injection, interpreter injection, locale/Unicode attacks, file system attacks, and buffer overflows.”

Just remember these words whenever you start writing APIs,

“All Input is Evil”

Does golang provide a nice way to validate the inputs? Let’s check it out.

The Naive Approach

When I started out writing in Go, I was mostly handling all inputs by my programming power. So I used to write something like this:

package main

import (
	"fmt"
	"regexp"
	"time"
)

type Order struct {
	OrderID          string    `json:"order_id"`
	CallbackURL      string    `json:"callback_url"`
	CustomerEmail    string    `json:"customer_email"`
	CustomerFullName string    `json:"customer_full_name"`
	Description      string    `json:"description"`
	Currency         string    `json:"currency"`
	ExpiredAt        time.Time `json:"expired_at"`
	InvoiceTime      string    `json:"invoice_time"`
	IP               string    `json:"ip"`
	MerchantOrderID  string    `json:"merchant_order_id"`
	OrderFrom        string    `json:"order_from"`
	OrderStatus      string    `json:"order_status"`
	ReceiveAmount    int       `json:"receive_amount"`
	Theme            string    `json:"theme"`
	Title            string    `json:"title"`
	UserID           string    `json:"user_id"`
}

func main() {
	o := Order{OrderID: "random_order_id", CallbackURL: "https://crazyurl.com", CustomerEmail: "crazygolang@go.com", IP: "222.222.222.222"}
	err := validateOrder(o)
	if err != nil {
		fmt.Println("Error Message: ", err.Error())
	}
}

func validateOrder(o Order) error {
	if !validateUUID(o.OrderID) {
		return fmt.Errorf("%s is not a valid uuid", o.OrderID)
	}
	if !validateURL(o.CallbackURL) {
		return fmt.Errorf("%s is not a valid URL", o.CallbackURL)
	}
	if !validateEmail(o.CustomerEmail) {
		return fmt.Errorf("%s is not a valid email", o.CustomerEmail)
	}
	if !validateIP(o.IP) {
		return fmt.Errorf("%s is not a valid IP", o.IP)
	}
	return nil
}

func validateURL(url string) bool {
	Re := regexp.MustCompile(`https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&//=]*)`)
	return Re.MatchString(url)
}

func validateUUID(uuid string) bool {
	Re := regexp.MustCompile(`[a-f0-9]{8}-?[a-f0-9]{4}-?4[a-f0-9]{3}-?[89ab][a-f0-9]{3}-?[a-f0-9]{12}`)
	return Re.MatchString(uuid)
}

func validateEmail(email string) bool {
	Re := regexp.MustCompile(`^[a-z0-9._%+\-]+@[a-z0-9.\-]+\.[a-z]{2,4}$`)
	return Re.MatchString(email)
}

func validateIP(ip string) bool {
	Re := regexp.MustCompile(`^(\d|[1-9]\d|1\d\d|2([0-4]\d|5[0-5]))\.(\d|[1-9]\d|1\d\d|2([0-4]\d|5[0-5]))\.(\d|[1-9]\d|1\d\d|2([0-4]\d|5[0-5]))\.(\d|[1-9]\d|1\d\d|2([0-4]\d|5[0-5]))$`)
	return Re.MatchString(ip)
}

Which is not bad when you’ve just started, it validates the input well enough and there is nothing wrong in taking this approach but as you can see there are just too many parameters to validate and it’s gonna take a lot of time writing(copy&paste) regex for each of the fields and it’s just not the idiomatic approach.

The Golang way

So we’ve talked about the naive approach which I still think is the best approach if you’re having enough time and experience dealing with regexs. You don’t have to depend on a library. Libraries/pre-written-modules are as much as harmful as the inputs so it’s always advised to choose your library carefully!! Well that’s a talk for another day. Let’s straight away dive into the code.

package main

import (
	"fmt"
	"time"
	"encoding/json"
	"reflect"
	"strings"
	"gopkg.in/go-playground/validator.v9"
)

type Order struct {
	OrderID          string    `json:"order_id" validate:"uuid4"`
	CallbackURL      string    `json:"callback_url" validate:"url"`
	CustomerEmail    string    `json:"customer_email,omitempty" validate:"omitempty,email"`
	CustomerFullName string    `json:"customer_full_name" validate:"min=3,max=30`
	Description      string    `json:"description" validate:"min=20,max=300"`
	Currency         string    `json:"currency" validate:"oneof=USD EUR AUD"`
	ExpiredAt        time.Time `json:"expired_at"`
	IP               string    `json:"ip" validate:"ipv4"`
	OrderStatus      string    `json:"order_status" validate:"oneof=pending completed failed processing"`
	ReceiveAmount    uint32    `json:"receive_amount,omitempty" validate:"gt=10,lt=10000"`
	Theme            string    `json:"theme" validate:"oneof=light dark"`
	Title            string    `json:"title" validate:"min=20,max=140"`
}

func main() {
	o := Order { OrderID: "f7f62726-b9a2-45f7-ad62-b3fe7eda8214", 
				CallbackURL: "https://crazyurl.com",
				CustomerFullName: "Bruha Nama",
				Description: "Lorem ipsum is lorem ipsum is again lorem ipsum then lorem ipsum",
				Currency: "EUR",
				IP: "222.222.222.255",
				OrderStatus: "pending",
				Theme: "light",
				Title: "Lorem ipsum is lorem ipsum is again lorem ipsum then lorem ipsum",
		}
	r, _ := json.Marshal(o)
	fmt.Println("Order", string(r))
	
	v := validator.New()

	err := v.Struct(o)
	if err != nil {
		for _, e := range err.(validator.ValidationErrors) {
			fmt.Printf(e)
			return
		}
	}
}

Yeah, you’re right. I’ve imported a beautiful library: go-playground / validator

and now you can see how easy it is to validate a field. You want to validate an uuid(version 4), just write

// uuid4: validates whether the order_id is in the right format.
json:"order_id" validate:"uuid4"

It validates only uuid4, if you want to validate any other version. Just change the value(eg. uuid, uuid5) and you’re good to go.

//omitempty : makes the field optional
//email: validates if the email is in the correct format.

json:"customer_email,omitempty" validate:"omitempty,email"

As you would’ve noticed that there are two “omitempty” in the code. The first one with json as key signifies that if there is no customer_email to validate, the final json you’re gonna get will be without customer_email. and the second one with validate as key signifies that the field is optional.

// url: validates whether the callback_url is valid.
json:"callback_url" validate:"url"

validation of URLs is now easy. Just write “url” inside validate and you’re done.

// min, max: characters in the description should be more than 20 and less than 300.
json:"description" validate:"min=20,max=300"

You need to validate a whole paragraph, use “min” , “max”.

// oneof: currency should be one of USD, EUR and AUD.
json:"currency" validate:"oneof=USD EUR AUD"

This is one of the interesting feature I found. If you’re having a field where you know the expected values, you can use “oneof” like in the example above. It accepts only USD, EUR and AUD.

// ipv4: validates whether ip is version 4.
json:"ip" validate:"ipv4"

IP validation is quite easy too. Just plug in the ip version you want to validate. or just “ip” if you are accepting ipv4 and ipv6 both.

Custom Validation

Till now we’ve fields where it was not necessary to customise the validation. let’s see how we can achieve this: Suppose I want to validate an url which starts with “https://”

type SecureURL struct {   
    URL `json:"url" validate:"customrl"`
}

v.RegisterValidation("customurl", func(fl validator.FieldLevel) bool {
    Re := regexp.MustCompile(`https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&//=]*)`)
    return Re.MatchString(url)
})

so what RegisterValidation is doing that it takes the url and matches with the regex if it passes it returns true otherwise false.

Error Handling

Now let’s look at the way we’re gonna handle the errors.

If any of the input fails, it’s gonna return the error like this:

Key: ‘Order.CustomerEmail’ Error:Field validation for ‘CustomerEmail’ failed on the ‘email’ tag

and you don’t wanna send this back to the client or the frontend guy. It is informative but it looks horrible as a message for the client.

So to overcome the above message, define a custom function as follows:

v.RegisterTagNameFunc(func(fld reflect.StructField) string {
    name := strings.SplitN(fld.Tag.Get("json"), ",", 2)[0]
    if name == "-" {
         return ""
    }
    return name
})

err := v.Struct(o)
    if err != nil {
        for _, e := range err.(validator.ValidationErrors) {
            fmt.Printf("Error Message: %s is invalid!!!", e.Field())
            return
    }
}

It gets the json field from the struct we defined earlier and if there is any error we’re gonna get the error like this:

Error Message: customer_email is invalid!!!

which is more presentable and your frontend guy will be happy too. You can also get the value of the field using e.Value().

I will try to explain some of the advance functionalities like slice validation, map validation in the next article.

Till then, Happy Coding!!!!