marmotte

marmotte is a modern gopher server.

What is marmotte?

marmotte is a modern way to run a modern gopher server. marmotte implements the Gopher protocol RFC1436, as well as the more recent extensions Gopher+ and the Gopher/Gopher+ URI Scheme RFC 4266. marmotte goes beyond this and also offers backwards-compatible ways to request code execution on the server. The possiblities include automatic conversion to any format, applying filters to data (images, text, audio...), offering a simple interface to batch jobs or toggling features in the server or even automatic redirection via request rewrite. marmotte is extensible and operations can be added dynamically by simply editing the configuration file.

Why bother?

Even though there is a fantastic wealth of information today on the Web, it always comes wrapped in a thick coat of advertisement, imposed aesthetics and user experience, and related and unrelated options all mixed together, from twitting to becoming a member to pinning on Pinterest to discussing on Reddit. This is not to say all these are bad in themselves, but rather that they tend to obfuscate or take precedence over actual content. An idea here is that the means of publication themselves have an influence on how the information is conveyed, and as a result, on how much information actually makes it to the user.

I believe that simpler means of publication yield simple means of use, that openness leads to creativity and that putting an emphasis on content attracts an audience with a taste and expectations for high quality content. This tends to create a virtuous circle, where the bar is constantly raised by the audience and, in turn, by the production, trying to meet and then exceed the expectations.

Also, it's fun. :)

An open design

marmotte is designed to support multiple protocols, and to be an easy to use all-in-one server. The focus today is on Gopher, but Gemini and other protocols will be implemented in the future (maybe even HTTP!).

The first protocol implemented by marmotte is Gopher. This decision is in direct connection to the technical qualities of the protocol: very straightforward, easy to implement, and amazingly open in its design, especially by today's standards. And this is also what motivates picking Gopher as a first implementation: Gopher embodies perfectly the spirit of the early Internet. Do not sell the information by diguising it, just share the information. Do not impose a specific design, let the users, via their client, decide for themselves. Do not impose, propose. Don't pretend, just stay genuine.

marmotte also will go beyond just serving local files. In the original RFC for Gopher, the protocol's mission is described as "distributed document search and retrieval". Once again, the loose, unassuming wording in the RFC opens the door to many interpretations, and one is to have Gopher work as a gateway to other protocols. This means marmotte can front remote sites running under a different protocol, like FTP, HTTP and NNTP, for example.

Getting started guide

To get started, simply clone this repository, edit main.go to set the root directory you want to serve, and run it.

$ git clone https://beastieboy.net/git/bestieboy/marmotte
$ cd marmotte
$ export MARMOTTE_ROOT=/path/to/your/gopher/root
$ go build -o marmotte cmd/marmotte/main.go
$ # you may want to place the binary somewhere on your path, like -o ~/bin/marmotte
$ ./marmotte # the server now listens on gopher://localhost:7070

Tour of the code

As of today, the code lives in a single package, simply called gopher. The code makes use of the following concepts and types:

• A Context contains the configuration for the running server, including the lists of Request and Response Transformers. It is the very first thing that is initialised, and is passed to all Request/Response-handling mechanism for reading and writing, updating the current status. Note that the Context is unique per server, not per Request, and should not be used to pass data from one Transformer down to another. Instances of Request and Response should be used for this purpose.
• A new Request is instantiated for each incoming connection that successfully sends a well-formatted Gopher request to the server. A Request contains information on its kind (today, Gophermap or simple path, tommorrow, functions and others). A Request is send down the series of all configured RequestTransformer instances (not implemented today, see below Limitations).
• A new Response instance is instantiated once a Request has been created. It contains information about the data requested, deduced from the MIME type of the file on the disk. This Response goes through the series of all configured ResponseTransformer instances before its content is sent back to the client that originated the request. The ResponseTransformer instances are responsible for preparing the data and applying all the modifications and adaptations necessary before the data is written back on the socket.
• A RequestTransformer is made of a TransformerPredicate and a RequestTransformerFunc. The former is a function that determines whether the RequestTransformer is applicable to the given Request and Context, and the latter is the function itself. Both the Request and the Context can be updated by the RequestTransformerFunc (a good example is a redirection, where the Request is updated to contain the new destination).
• A ResponseTransformer is made of a TransformerPredicate and a ResponseTransformerFunc. The tandem work the same way as for responses, where the former determines whether the latter is applicable to the given Context, Request and Response. Here, the Request has become read-only and only the Context and the Response instances can be modified. The various content sections of the Response instance are typically updated by the ResponseTransformerFunc.
• The file files.go is where file types are determined, based on the MIME type deduction.
• The file server.go contains the main loop, where network connections are accepted, parsed, transformations are applied and data is written back on the socket.
• The file errorcodes.go is where the gopher error types and codes are defined.

Usage

marmotte accepts a series of options on the command line to customise many aspects. Calling the server with the option --help gives the complete list:

$ marmotte --help
Usage of ./marmotte:
      --configfile string          YAML configuration file (default "/usr/local/etc/marmotte/marmotte.yaml")
      --debug                      set logging level to Debug
      --errors string              the directory where the error templates can be found (default "/errors")
      --host string                the hostname the server sockets will bind to (default "localhost")
      --logfile string             the logfile (not rotated) (default "/var/log/marmotte.log")
      --port int                   the port the server will listen on (default 7070)
      --root string                the root directory where the documents can be found (default "/usr/local/marmotte")
      --transport string           the network transport protocol to use for sockets (default "tcp")
      --users_gopherspace string   the root directory for users gopherspace (~username) (default "/users")

The table below explains explains each option in detail:

Option	Description	Examples
host	The hostname the sockets created by the server will bind to. The server will then be accessible via the address host:port.	localhost mygopherhole.net
port	The port number the socket created by the server will bind to. The server will then be accessible via the address host:port.	7070 70 (if running as a priviledged user on Unix-like operating systems)
root	The root directory where all the resources can be found. This includes all the files, directories and the programs and scripts. The directory must be at least read-accessible. In case executables are called by the server, the subdirectory tmp must be write-accessible to the operating system user or group the server belongs to.	/usr/local/marmotte /data/
transport	The network transport protocol to be used by the server. Today, only tcp makes sense here.	tcp
users_gopherspace	marmotte offers users' gopherspace (refer to the section Users' gopherspace for details on what it is and directions to set this up). This option allows to specify the root directory for the users' gopherspace, which will be resolved as a subdirectory of root.	users_gopherspace homes
errors	Root directory where the error content templates can be found. For more information on error templates and error handling in general, please refer to the section Error handling. The directory is resolved as a subdirectory of root.	errors custom/error_templates
logfile	The full path to the logfile. If none is given, the logs are printed on stdout, which allows to have a logshipper take care of them. It is recommended the logfile lives outside of root.	/var/log/marmotte.log
debug	Flag to toggle debug messages in the logs on and off	true false
configfile	The path to the configuration file for marmotte, relative to root. Refer to the section Configuring marmotte for details.	/usr/local/etc/marmotte/marmotte.yaml

Configuring marmotte

marmotte can be configured via a YAML file, environment variables and the command line. marmotte searches for settings and applies them going through these sources in this order. This allows to override a parameter defined in a configuration file by passing an option on the command line, for example. The environment variables are useful when deploying in a container and in a cloud context, typically.

The preceding section presents all the options available on the command line.

The name of the different environment variables are exactly as the command-line options, with the prefix MARMOTTE_. For example, the --root option's counterpart in the environment is MARMOTTE_ROOT. These environment variables can be defined in a .env file placed next to the binary, or can be defined and exported directly in the environment.

The configuration file uses also the exact same names as the command line options. Below is an example of such a configuration file:

# Example configuration file for marmotte
# Copy and edit to suits your needs

# Host the sockets will be bound to
host: localhost

# Port the sockets will be bound to
# Note: On Unix-like systems, to listen to ports below 1024 a process must run as a privileged user
port: 7070

# Network transport protocol the server will create the sockets with
# Today, only tcp makes sense here
transport: tcp

# The directory where gopher resources can be found
# The directory must be readable by the user running the server
root: /usr/local/marmotte

# When not shipping logs to an aggregator, set this to a writable file
# To print on stdout and/or use a logshipper, comment out the following line
logfile: /var/log/marmotte

# The directory alias for users' gopherspaces
# The path is relative to the root directory
users_gopherspace: /users

# The path to the directory containing the error templates (10.gt, 20.gt, 30.gt, etc.)
# The path is relative to the root directory
errors: /errors

# Whether log level should be set to debug
debug: false

Customising gophermaps and text files

marmotte offers a simple way to customise gophermaps and text files via Headers and Footers. The Context holds Header and Footer data as a slice of strings in a map. By combining Predicates and the functions HeaderTransformerfor and FooterTransformerfor, one can define Headers only applicable to specific situations, and bind specific Headers and Footers to these situations.

The code in cmd/marmotte/main.go gives a good example:

	// custom header for gophermaps
	ctx.Headers = map[string][]string {
		"all_gophermaps": { "Welcome to " + ctx.Host,
			"This gopherhole is powered by marmotte"},
	}
	
	ctx.ResponseTransformers = append(ctx.ResponseTransformers, gopher.ResponseTransformer{
		Transformer: gopher.HeaderTransformerFor("all_gophermaps"),
		Predicate: gopher.GopherMapPredicate })

In the code above, a simple Header is defined under the key "all_gophermaps". There are no special keywords here; it can be anything. Then, a ResponseTransformer is added to the Context that applies the Headers to a Response (prepends the text to the content) found behind that same key, "all_gophermaps", and reserve its use to gophermaps by tying it with the Predicate GopherMapPredicate. That way, text files will not come with this Header, only gophermaps.

Another way to customise gophermaps is to write dynamic gophermaps. See the section Dynamic content for details.

Executables

marmotte allows executables to be called and returns the result to the client. The executables can be of any nature: as long as their permissions in the filesystem allow the operating system user or group marmotte runs as to execute them, they represent valid resources.

The executables must return a single string, which should be either a path to a file, or a path to a directory. When generating data, the data should be placed in a file or in a collection of files. No data should be printed on stdout or stderr. It is generally a good idea to wrap an external program in a shell script and to make this script callable by marmotte. Files should be placed under the directory $root/tmp. The file testdata/createtextfile.sh is an example of such a shell script.

The executable should be referenced in a gophermap by giving the type of the file or files generated: for example, 0 for a text file, g for a graphics file, 1 for a directory (in case a collection of files are generated). For example:

0Generate a text file    /createtextfile.sh    mygopherhole.net    70
1Generate a collection of images    /image-gen    mygopherhole.net    70

Dynamic content

marmotte is able to generate dynamic content via a templating system. A ResponseTransformer handles the processing of the templates, and returns plain text content. The syntax for these templates is detailed in the page that documents the Go module text/template.

marmotte today passes the following struct to all templates:

type DynamicContext struct {
	ServerContext Context
	ErrorInfo GopherErrorInfo
	RequestInfo Request
	ResponseInfo Response
	CurrentTime time.Time
}

Below is an example of a template using this structure:

Hello there! This content has been dynamically generated.

Current Configuration:
  - Server Context
    - Host: {{- .ServerContext.Host }}
    - Port: {{- .ServerContext.Port }}
    - TransportProtocol: {{- .ServerContext.TransportProtocol }}
    - Root: {{- .ServerContext.Root }}
    - ErrorRoot: {{- .ServerContext.ErrorRoot }}
    - UsersGopherspace: {{- .ServerContext.UsersGopherspace }}

  - Request
    - FullPath: {{- .RequestInfo.FullPath }}
    - Path: {{- .RequestInfo.Path }}
  - Additional parameters
    - Current time: {{- .CurrentTime }}

In the future, this structure will offer more data to the templates to play with.

Error handling

marmotte offers dynamic error handling, similarly to what is found usually in http servers. Depending on the error encountered, a different message is reported to the user, with some information to help understand the error. As the Gopher Protocol leaves all the details of error handling to the server implementation, there are no standard error codes or specific mechanism expected by Gopher clients. marmotte handles error by redirecting a failed request to some content informing the user of the problem. This content is a template found in the errors subdirectory of root. There is one template per error code. Today, the templates are in plain text. marmotte defines the following error codes and templates:

Error code	Description	Template
00	No error. This code is used when controlling the dispatching of the ResponseTransformers, to allow a pipeline to continue.	No template (normal response returned)
10	Unrecognised selector. This code is used whenever a selector has been submitted that, after going the RequestTransformer and ResponseTransformer pipelines hasn't been picked up by any ResponseTransformer.	$root/$errors/10.gt
20	Wrong permissions. This code is used whenever the server cannot access the resources behind a given selector.	$root/$errors/20.gt
30	Could not retrieve data. This error code is used whenever an error occur while reading the contents of a file. It is also used whenever an error template itself cannot be read.	$root/$errors/30.gt

Error handling is done via a ResponseTransformer that comes with marmotte and that is set up by default.

Customising error messages

marmotte comes with a set of default templates currently located in testdata/errors/. These files are Go templates and can be placed in $root/$errors and modified there. marmotte passes an instance of GopherErrorInfo to the templates. This data type is defined in the file errorcodes.go.

Virtual folders and selector rewriting

marmotte offers selector rewriting facilities in the form of a RequestTransformer. The function SelectorRewriteTransformerFor takes a string from and a string to and returns a RequestTransformerFunc that replaces occurrences of from with to in the Path and FullPath fields of a Request instance. Both strings can be regular expressions, and to can reference matches found against from in the Request instance's FullPath via the usual $1, $2, etc.

Users' gopherspace

marmotte offers the possibility for multiple users to have their own gopherspace. This is achieved through selector rewriting. The individual gopherspaces are placed inside the directory $root/$users_gopherspace, and should be named after the users. They become available through tilde selectors, such as:

1Spock's gopher!    ~spock    enterprisesub.space    70
1Picard's phlog    /~picard    enterprisesub.space    70

The extra / in front of the selector is optional, and these two selectors will resolve as $root/$users_gopherspace/spock and $root/$users_gopherspace/picard, both typically directories, and as such will display as a gophermap.

Limitations

marmotte is under active development, and currently does not implement all the features listed in the first section, What is marmotte?.

Currently implemented features

Below is a list of the features currently implemented. This is by no means a complete roadmap, but gives an overview of the usability of marmotte for a given purpose.

• Gopher Protocol RFC 1436
• Gopher+
• Response Transformers infrastructure
• Gopher protocol implemented via Transformers
• Gophermap Header Transformer
• Backwards-compatible selector/URI to offer and select Transformers
• Request Transformers infrastructure
• Simple redirect based on Request Transformers
• Dynamic error content via templates
• Dynamic content via templates
• Dynamic execution of scripts and programs
• Image filtering via Transformers
• Search

License

marmotte is published under the 2-Clause BSD License. A copy of the license is included in the source repository.