From Docker to Kubernetes, from Consul to Terraform, Go has been used increasingly in system tools these last years.
Since most of these system tools manage systems running on Unix systems, one of their core tasks is to deal with files, and configuration files in particular.
Augeas: the configuration management scalpel
Augeas is a C library to modify configuration files. It allows to parse files with many different syntax (over 300 by default), modify the configuration using a tree accessed with an XPath-like language, and write back the configuration.
It tries hard to modify only what you mean to, keeping all details (spaces, indentations, new lines, comments) unchanged.
Because of its history, Augeas is mainly known in the Puppet world. However, there are also plugins for Ansible, Chef, SaltStack, (R)?ex and more tools… Augeas is also used directly in C libraries such as libvirt and Nut.
Augeasproviders
In the Puppet world, the Augeasproviders project was created to develop native Puppet types and providers (in Ruby) based on Augeas.
These providers use the Augeas Ruby bindings to draw on Augeas' power, all the while providing a simple interface for users, without the need to know how Augeas works.
At the core of the Augeasproviders project, there is a base provider shipped in the hearculesteam-augeasproviders_core Puppet module, which provides an interface to build more providers, in a declarative way.
For example, you can set the location of the node corresponding to the Puppet resource to manage in the Augeas tree:
resource_pathdo|resource|service=resource[:service]type=resource[:type]mod=resource[:module]control_cond=(resource[:control_is_param]==:true)?"and
control='#{resource[:control]}'":''iftarget=='/etc/pam.conf'"$target/*[service='#{service}' and type='#{type}' and module='#{mod}' #{control_cond}]"else"$target/*[type='#{type}' and module='#{mod}' #{control_cond}]"endend
The create and destroy methods, as well as the getters and setters for the Puppet resource properties, can also be described in a similar fashion, making it simpler to develop new providers based on Augeas.
Go bindings
As for many other languages, there are Go bindings for Augeas:
Much like the Ruby bindings, the go library lets you manipulate an Augeas handler to query the Augeas tree, modify it, and save it.
Go structure tags
In the Go world, structures have optional tags which can be used for parsing and writing to external formats.
This is used to reflect structures as JSON, YAML, XML, or specify library options to manage the structure fields:
// Version is an S3 bucket versiontypeVersionstruct{IDuint`sql:"AUTO_INCREMENT" gorm:"primary_key" json:"-"`VersionIDstring`gorm:"index" json:"version_id"`LastModifiedtime.Time`json:"last_modified"`}
typeconfigstruct{Versionbool`short:"V" long:"version" description:"Display version."`Tokenstring`short:"t" long:"token" description:"GitHub token" env:"GITHUB_TOKEN"`Users[]string`short:"u" long:"users" description:"GitHub users to include (comma separated)." env:"GITHUB_USERS" env-delim:","`Manpagebool`short:"m" long:"manpage" description:"Output manpage."`}
The tags above (sql, gorm, json, short, long, description, env, env-delim) are used by Go libraries through the Go reflection library to provide dynamic features for structures.
Narcissus: Augeasproviders for the Go world
While Hercules is known in Greek mythology for his works —including cleaning the stables of King Augeas—, Narcissus is famous for gazing at his reflection in the water.
The Narcissus project is a Go library providing structure tags to manage configuration files with Augeas. It then maps structure tags to the Augeas tree dynamically, allowing you to expose any configuration file (or file stanza) known to Augeas as a Go structure.
Example of /etc/group
The Unix group file is very simple and well-known. It features one group per line, with fields separated by colons:
Augeas parses it by storing each group name as a node key in the tree, and exposing each field by its name:
$augtool print /files/etc/group
/files/etc/group
/files/etc/group/root
/files/etc/group/root/password = "x"
/files/etc/group/root/gid = "0"
/files/etc/group/daemon
/files/etc/group/daemon/password = "x"
/files/etc/group/daemon/gid = "1"
/files/etc/group/bin
/files/etc/group/bin/password = "x"
/files/etc/group/bin/gid = "2"
/files/etc/group/sys
/files/etc/group/sys/password = "x"
/files/etc/group/sys/gid = "3"
/files/etc/group/adm
/files/etc/group/adm/password = "x"
/files/etc/group/adm/gid = "4"
/files/etc/group/adm/user[1] = "syslog"
/files/etc/group/adm/user[2] = "raphink"
Modifying any of these fields and saving the tree will result in an updated /etc/group file. Adding new entries in the tree will result in additional entries in /etc/group, provided the tree is valid for the Group.lns Augeas lens.
Parsing with Narcissus
In our Go code, we can map a group structure to entries in the /etc/group file easily by using the Narcissus package:
import("log""honnef.co/go/augeas""github.com/raphink/narcissus")typegroupstruct{augeasPathstringNamestring`narcissus:".,value-from-label"`Passwordstring`narcissus:"password"`GIDint`narcissus:"gid"`Users[]string`narcissus:"user"`}funcmain(){aug,err:=augeas.New("/","",augeas.None)iferr!=nil{log.Fatal("Failed to create Augeas handler")}n:=narcissus.New(&aug)group:=&group{augeasPath:"/files/etc/group/docker",}err=n.Parse(group)iferr!=nil{log.Fatalf("Failed to retrieve group: %v",err)}log.Printf("GID=%v",group.GID)log.Printf("Users=%v",strings.Join(group.Users,","))}
The augeasPath field is necessary to store the location of the file in the Augeas tree, in our case /files/etc/group/docker to manage the docker group in the file.
Then each structure field is linked to the corresponding node name in the Augeas tree:
Name is taken from the node label, so we use the special value .,value-from-label, where . refers to the current node, and value-from-label tells Narcissus how to get the value
password for the Password
gid for the GID
user for the Users, parsed as a slice of strings (i.e. the user label might appear more than once in the Augeas tree)
Note that all fields must be capitalized in order for Go reflection to work.
Once we call the Parse() method on the Narcissus handler, the structure is dynamically filled with the values in the Augeas tree, so we can access the gid with group.GID and the users with group.Users.
Modifying files
The main point of the Augeas library is not just to parse, but also to modify configuration files in a versatile way.
In Narcissus, this is done by calling the Write() method on the Narcissus handler. Narcissus then transforms the structure back to the Augeas tree and saves it.
For example, using the PasswdUser type provided by default in the narcissus package:
user:=n.NewPasswdUser("raphink")// Modify UIDuser.UID=42iferr:=n.Write(user);err!=nil{log.Fatalf("Failed to save user: %v",err)}
Included formats
Narcissus comes with a few structures already mapped:
/etc/fstab, with the NewFstab() method
/etc/hosts with the NewHosts() method
/etc/passwd with NewPasswd() and NewPasswdUser() methods
/etc/services with NewServices() and NewService() methods
Which structures will you map with it? Which tool could benefit from this library?