|
1 year ago | |
---|---|---|
cmd/marmotte | 1 year ago | |
configs | 1 year ago | |
gopher | 1 year ago | |
testdata | 1 year ago | |
.gitignore | 1 year ago | |
LICENSE | 1 year ago | |
Makefile | 1 year ago | |
README.md | 1 year ago | |
go.mod | 1 year ago | |
go.sum | 1 year ago | |
gopherplus.txt | 1 year ago |
README.md
marmotte
marmotte is a modern gopher server.
What is marmotte?
marmotte is a modern way to run a modern gopher server. marmotte implements the Gopher protocol RFC1436, as well as the more recent extensions Gopher+ and the Gopher/Gopher+ URI Scheme RFC 4266. marmotte goes beyond this and also offers backwards-compatible ways to request code execution on the server. The possiblities include automatic conversion to any format, applying filters to data (images, text, audio...), offering a simple interface to batch jobs or toggling features in the server or even automatic redirection via request rewrite. marmotte is extensible and operations can be added dynamically by simply editing the configuration file.
Why bother?
Even though there is a fantastic wealth of information today on the Web, it always comes wrapped in a thick coat of advertisement, imposed aesthetics and user experience, and related and unrelated options all mixed together, from twitting to becoming a member to pinning on Pinterest to discussing on Reddit. This is not to say all these are bad in themselves, but rather that they tend to obfuscate or take precedence over actual content. An idea here is that the means of publication themselves have an influence on how the information is conveyed, and as a result, on how much information actually makes it to the user.
I believe that simpler means of publication yield simple means of use, that openness leads to creativity and that putting an emphasis on content attracts an audience with a taste and expectations for high quality content. This tends to create a virtuous circle, where the bar is constantly raised by the audience and, in turn, by the production, trying to meet and then exceed the expectations.
Also, it's fun. :)
An open design
marmotte is designed to support multiple protocols, and to be an easy to use all-in-one server. The focus today is on Gopher, but Gemini and other protocols will be implemented in the future (maybe even HTTP!).
The first protocol implemented by marmotte is Gopher. This decision is in direct connection to the technical qualities of the protocol: very straightforward, easy to implement, and amazingly open in its design, especially by today's standards. And this is also what motivates picking Gopher as a first implementation: Gopher embodies perfectly the spirit of the early Internet. Do not sell the information by diguising it, just share the information. Do not impose a specific design, let the users, via their client, decide for themselves. Do not impose, propose. Don't pretend, just stay genuine.
marmotte also will go beyond just serving local files. In the original RFC for Gopher, the protocol's mission is described as "distributed document search and retrieval". Once again, the loose, unassuming wording in the RFC opens the door to many interpretations, and one is to have Gopher work as a gateway to other protocols. This means marmotte can front remote sites running under a different protocol, like FTP, HTTP and NNTP, for example.
Getting started guide
To get started, simply clone this repository, edit main.go
to set the root directory you want to serve, and run it.
$ git clone https://beastieboy.net/git/bestieboy/marmotte
$ cd marmotte
$ export MARMOTTE_ROOT=/path/to/your/gopher/root
$ go build -o marmotte cmd/marmotte/main.go
$ # you may want to place the binary somewhere on your path, like -o ~/bin/marmotte
$ ./marmotte # the server now listens on gopher://localhost:7070
Tour of the code
As of today, the code lives in a single package, simply called gopher
. The code makes use of the following concepts and types:
- A
Context
contains the configuration for the running server, including the lists of Request and Response Transformers. It is the very first thing that is initialised, and is passed to all Request/Response-handling mechanism for reading and writing, updating the current status. Note that theContext
is unique per server, not per Request, and should not be used to pass data from one Transformer down to another. Instances ofRequest
andResponse
should be used for this purpose. - A new
Request
is instantiated for each incoming connection that successfully sends a well-formatted Gopher request to the server. ARequest
contains information on its kind (today, Gophermap or simple path, tommorrow, functions and others). ARequest
is send down the series of all configuredRequestTransformer
instances (not implemented today, see below Limitations). - A new
Response
instance is instantiated once aRequest
has been created. It contains information about the data requested, deduced from the MIME type of the file on the disk. ThisResponse
goes through the series of all configuredResponseTransformer
instances before its content is sent back to the client that originated the request. TheResponseTransformer
instances are responsible for preparing the data and applying all the modifications and adaptations necessary before the data is written back on the socket. - A
RequestTransformer
is made of aTransformerPredicate
and aRequestTransformerFunc
. The former is a function that determines whether theRequestTransformer
is applicable to the givenRequest
andContext
, and the latter is the function itself. Both theRequest
and theContext
can be updated by theRequestTransformerFunc
(a good example is a redirection, where theRequest
is updated to contain the new destination). - A
ResponseTransformer
is made of aTransformerPredicate
and aResponseTransformerFunc
. The tandem work the same way as for responses, where the former determines whether the latter is applicable to the givenContext
,Request
andResponse
. Here, theRequest
has become read-only and only theContext
and theResponse
instances can be modified. The various content sections of theResponse
instance are typically updated by theResponseTransformerFunc
. - The file
files.go
is where file types are determined, based on the MIME type deduction. - The file
server.go
contains the main loop, where network connections are accepted, parsed, transformations are applied and data is written back on the socket. - The file
errorcodes.go
is where the gopher error types and codes are defined.
Usage
marmotte accepts a series of options on the command line to customise many aspects. Calling the server with the option --help
gives the complete list:
$ marmotte --help
Usage of ./marmotte:
--configfile string YAML configuration file (default "/usr/local/etc/marmotte/marmotte.yaml")
--debug set logging level to Debug
--errors string the directory where the error templates can be found (default "/errors")
--host string the hostname the server sockets will bind to (default "localhost")
--logfile string the logfile (not rotated) (default "/var/log/marmotte.log")
--port int the port the server will listen on (default 7070)
--root string the root directory where the documents can be found (default "/usr/local/marmotte")
--transport string the network transport protocol to use for sockets (default "tcp")
--users_gopherspace string the root directory for users gopherspace (~username) (default "/users")
The table below explains explains each option in detail:
Option | Description | Examples |
---|---|---|
host | The hostname the sockets created by the server will bind to. The server will then be accessible via the address host:port . |
localhost mygopherhole.net |
port | The port number the socket created by the server will bind to. The server will then be accessible via the address host:port . |
7070 70 (if running as a priviledged user on Unix-like operating systems) |
root | The root directory where all the resources can be found. This includes all the files, directories and the programs and scripts. The directory must be at least read-accessible. In case executables are called by the server, the subdirectory tmp must be write-accessible to the operating system user or group the server belongs to. |
/usr/local/marmotte /data/ |
transport | The network transport protocol to be used by the server. Today, only tcp makes sense here. |
tcp |
users_gopherspace | marmotte offers users' gopherspace (refer to the section Users' gopherspace for details on what it is and directions to set this up). This option allows to specify the root directory for the users' gopherspace, which will be resolved as a subdirectory of root . |
users_gopherspace homes |
errors | Root directory where the error content templates can be found. For more information on error templates and error handling in general, please refer to the section Error handling. The directory is resolved as a subdirectory of root . |
errors custom/error_templates |
logfile | The full path to the logfile. If none is given, the logs are printed on stdout , which allows to have a logshipper take care of them. It is recommended the logfile lives outside of root . |
/var/log/marmotte.log |
debug | Flag to toggle debug messages in the logs on and off | true false |
configfile | The path to the configuration file for marmotte, relative to root . Refer to the section Configuring marmotte for details. |
/usr/local/etc/marmotte/marmotte.yaml |
Configuring marmotte
marmotte can be configured via a YAML file, environment variables and the command line. marmotte searches for settings and applies them going through these sources in this order. This allows to override a parameter defined in a configuration file by passing an option on the command line, for example. The environment variables are useful when deploying in a container and in a cloud context, typically.
The preceding section presents all the options available on the command line.
The name of the different environment variables are exactly as the command-line options, with the prefix MARMOTTE_
. For example, the --root
option's counterpart in the environment is MARMOTTE_ROOT
. These environment variables can be defined in a .env
file placed next to the binary, or can be defined and exported directly in the environment.
The configuration file uses also the exact same names as the command line options. Below is an example of such a configuration file:
# Example configuration file for marmotte
# Copy and edit to suits your needs
# Host the sockets will be bound to
host: localhost
# Port the sockets will be bound to
# Note: On Unix-like systems, to listen to ports below 1024 a process must run as a privileged user
port: 7070
# Network transport protocol the server will create the sockets with
# Today, only tcp makes sense here
transport: tcp
# The directory where gopher resources can be found
# The directory must be readable by the user running the server
root: /usr/local/marmotte
# When not shipping logs to an aggregator, set this to a writable file
# To print on stdout and/or use a logshipper, comment out the following line
logfile: /var/log/marmotte
# The directory alias for users' gopherspaces
# The path is relative to the root directory
users_gopherspace: /users
# The path to the directory containing the error templates (10.gt, 20.gt, 30.gt, etc.)
# The path is relative to the root directory
errors: /errors
# Whether log level should be set to debug
debug: false
Customising gophermaps and text files
marmotte offers a simple way to customise gophermaps and text files via Headers and Footers. The Context holds Header and Footer data as a slice of strings in a map. By combining Predicates
and the functions HeaderTransformerfor
and FooterTransformerfor
, one can define Headers only applicable to specific situations, and bind specific Headers and Footers to these situations.
The code in cmd/marmotte/main.go
gives a good example:
// custom header for gophermaps
ctx.Headers = map[string][]string {
"all_gophermaps": { "Welcome to " + ctx.Host,
"This gopherhole is powered by marmotte"},
}
ctx.ResponseTransformers = append(ctx.ResponseTransformers, gopher.ResponseTransformer{
Transformer: gopher.HeaderTransformerFor("all_gophermaps"),
Predicate: gopher.GopherMapPredicate })
In the code above, a simple Header is defined under the key "all_gophermaps". There are no special keywords here; it can be anything. Then, a ResponseTransformer
is added to the Context that applies the Headers to a Response (prepends the text to the content) found behind that same key, "all_gophermaps", and reserve its use to gophermaps by tying it with the Predicate
GopherMapPredicate
. That way, text files will not come with this Header, only gophermaps.
Another way to customise gophermaps is to write dynamic gophermaps. See the section Dynamic content for details.
Executables
marmotte allows executables to be called and returns the result to the client. The executables can be of any nature: as long as their permissions in the filesystem allow the operating system user or group marmotte runs as to execute them, they represent valid resources.
The executables must return a single string, which should be either a path to a file, or a path to a directory. When generating data, the data should be placed in a file or in a collection of files. No data should be printed on stdout
or stderr
. It is generally a good idea to wrap an external program in a shell script and to make this script callable by marmotte. Files should be placed under the directory $root/tmp
. The file testdata/createtextfile.sh
is an example of such a shell script.
The executable should be referenced in a gophermap by giving the type of the file or files generated: for example, 0 for a text file, g for a graphics file, 1 for a directory (in case a collection of files are generated). For example:
0Generate a text file /createtextfile.sh mygopherhole.net 70
1Generate a collection of images /image-gen mygopherhole.net 70
Dynamic content
marmotte is able to generate dynamic content via a templating system. A ResponseTransformer
handles the processing of the templates, and returns plain text content. The syntax for these templates is detailed in the page that documents the Go module text/template.
marmotte today passes the following struct to all templates:
type DynamicContext struct {
ServerContext Context
ErrorInfo GopherErrorInfo
RequestInfo Request
ResponseInfo Response
CurrentTime time.Time
}
Below is an example of a template using this structure:
Hello there! This content has been dynamically generated.
Current Configuration:
- Server Context
- Host: {{- .ServerContext.Host }}
- Port: {{- .ServerContext.Port }}
- TransportProtocol: {{- .ServerContext.TransportProtocol }}
- Root: {{- .ServerContext.Root }}
- ErrorRoot: {{- .ServerContext.ErrorRoot }}
- UsersGopherspace: {{- .ServerContext.UsersGopherspace }}
- Request
- FullPath: {{- .RequestInfo.FullPath }}
- Path: {{- .RequestInfo.Path }}
- Additional parameters
- Current time: {{- .CurrentTime }}
In the future, this structure will offer more data to the templates to play with.
Error handling
marmotte offers dynamic error handling, similarly to what is found usually in http servers. Depending on the error encountered, a different message is reported to the user, with some information to help understand the error. As the Gopher Protocol leaves all the details of error handling to the server implementation, there are no standard error codes or specific mechanism expected by Gopher clients. marmotte handles error by redirecting a failed request to some content informing the user of the problem. This content is a template found in the errors
subdirectory of root
. There is one template per error code. Today, the templates are in plain text. marmotte defines the following error codes and templates:
Error code | Description | Template |
---|---|---|
00 | No error. This code is used when controlling the dispatching of the ResponseTransformers, to allow a pipeline to continue. | No template (normal response returned) |
10 | Unrecognised selector. This code is used whenever a selector has been submitted that, after going the RequestTransformer and ResponseTransformer pipelines hasn't been picked up by any ResponseTransformer. | $root/$errors/10.gt |
20 | Wrong permissions. This code is used whenever the server cannot access the resources behind a given selector. | $root/$errors/20.gt |
30 | Could not retrieve data. This error code is used whenever an error occur while reading the contents of a file. It is also used whenever an error template itself cannot be read. | $root/$errors/30.gt |
Error handling is done via a ResponseTransformer
that comes with marmotte and that is set up by default.
Customising error messages
marmotte comes with a set of default templates currently located in testdata/errors/
. These files are Go templates and can be placed in $root/$errors
and modified there. marmotte passes an instance of GopherErrorInfo
to the templates. This data type is defined in the file errorcodes.go
.
Virtual folders and selector rewriting
marmotte offers selector rewriting facilities in the form of a RequestTransformer. The function SelectorRewriteTransformerFor
takes a string from
and a string to
and returns a RequestTransformerFunc
that replaces occurrences of from
with to
in the Path and FullPath fields of a Request instance. Both strings can be regular expressions, and to
can reference matches found against from
in the Request instance's FullPath via the usual $1
, $2
, etc.
Users' gopherspace
marmotte offers the possibility for multiple users to have their own gopherspace. This is achieved through selector rewriting. The individual gopherspaces are placed inside the directory $root/$users_gopherspace
, and should be named after the users. They become available through tilde selectors, such as:
1Spock's gopher! ~spock enterprisesub.space 70
1Picard's phlog /~picard enterprisesub.space 70
The extra /
in front of the selector is optional, and these two selectors will resolve as $root/$users_gopherspace/spock
and $root/$users_gopherspace/picard
, both typically directories, and as such will display as a gophermap.
Limitations
marmotte is under active development, and currently does not implement all the features listed in the first section, What is marmotte?.
Currently implemented features
Below is a list of the features currently implemented. This is by no means a complete roadmap, but gives an overview of the usability of marmotte for a given purpose.
- Gopher Protocol RFC 1436
- Gopher+
- Response Transformers infrastructure
- Gopher protocol implemented via Transformers
- Gophermap Header Transformer
- Backwards-compatible selector/URI to offer and select Transformers
- Request Transformers infrastructure
- Simple redirect based on Request Transformers
- Dynamic error content via templates
- Dynamic content via templates
- Dynamic execution of scripts and programs
- Image filtering via Transformers
- Search
License
marmotte is published under the 2-Clause BSD License. A copy of the license is included in the source repository.