Update documentation (#28)

Documentation updates:
- [X] Section on command line options
- [X] Section on dynamic error handling
- [X] Section on error templates
- [X] Section on users' gopherspace
- [X] Section on selector rewriting and virtual directories
- [X] API reference

Closes #26

Co-authored-by: Nicolas Herry <beastieboy@beastieboy.net>
Reviewed-on: #28
pull/31/head
beastieboy 1 year ago
parent 751b614ceb
commit ef0d90fb6c
  1. 86
      README.md
  2. 5
      gopher/context.go
  3. 6
      gopher/predicates.go
  4. 31
      gopher/request.go
  5. 2
      gopher/server.go
  6. 2
      gopher/transformers.go

@ -40,6 +40,85 @@ As of today, the code lives in a single package, simply called `gopher`. The cod
- A `ResponseTransformer` is made of a `TransformerPredicate` and a `ResponseTransformerFunc`. The tandem work the same way as for responses, where the former determines whether the latter is applicable to the given `Context`, `Request` and `Response`. Here, the `Request` has become read-only and only the `Context` and the `Response` instances can be modified. The various content sections of the `Response` instance are typically updated by the `ResponseTransformerFunc`.
- The file `files.go` is where file types are determined, based on the MIME type deduction.
- The file `server.go` contains the main loop, where network connections are accepted, parsed, transformations are applied and data is written back on the socket.
- The file `errorcodes.go` is where the gopher error types and codes are defined.
## Usage
marmotte accepts a series of options on the command line to customise many aspects. Calling the server with the option `--help` gives the complete list:
```sh
$ marmotte --help
Usage of /usr/local/bin/marmotte:
--errors string the directory where the error templates can be found (default "$ROOT/errors")
--host string the hostname the server sockets will bind to (default "localhost")
--logfile string the logfile (not rotated) (default "/var/log/marmotte.log")
--port int the port the server will listen on (default 7070)
--root string the root directory where the documents can be found (default "/usr/local/marmotte")
--transport string the network transport protocol to use for sockets (default "tcp")
--users_gopherspace string the root directory for users gopherspace (~username) (default "/users")
```
The table below explains explains each option in detail:
|Option|Description|Examples|
|---|:---:|---|
|host|The hostname the sockets created by the server will bind to. The server will then be accessible via the address `host:port`.|`localhost`<br>`mygopherhole.net`|
|port|The port number the socket created by the server will bind to. The server will then be accessible via the address `host:port`.|`7070`<br>`70` (if running as a priviledged user on Unix-like operating systems)|
|root|The root directory where all the resources can be found. This includes all the files, directories and the programs and scripts. The directory must be at least read-accessible. In case executables are called by the server, the subdirectory `tmp` must be write-accessible to the operating system user or group the server belongs to.|`/usr/local/marmotte`<br>`/data/`|
|transport|The network transport protocol to be used by the server. Today, only `tcp` makes sense here.|`tcp`|
|users_gopherspace|marmotte offers users' gopherspace (refer to the section [Users' gopherspace](#users-gopherspace) for details on what it is and directions to set this up). This option allows to specify the root directory for the users' gopherspace, which will be resolved as a subdirectory of `root`.|`users_gopherspace`<br>`homes`|
|errors|Root directory where the error content templates can be found. For more information on error templates and error handling in general, please refer to the section [Error handling](#error-handling). The directory is resolved as a subdirectory of `root`.|`errors`<br>`custom/error_templates`|
|logfile|The full path to the logfile. If none is given, the logs are printed on `stdout`, which allows to have a logshipper take care of them. It is recommended the logfile lives outside of `root`.|`/var/log/marmotte.log`|
## Customising gophermaps and text files
marmotte offers a simple way to customise gophermaps and text files via Headers and Footers. The Context holds Header and Footer data as a slice of strings in a map. By combining `Predicates` and the functions `HeaderTransformerfor` and `FooterTransformerfor`, one can define Headers only applicable to specific situations, and bind specific Headers and Footers to these situations.
The code in `cmd/marmotte/main.go` gives a good example:
```go
// custom header for gophermaps
ctx.Headers = map[string][]string {
"all_gophermaps": { "Welcome to " + ctx.Host,
"This gopherhole is powered by marmotte"},
}
ctx.ResponseTransformers = append(ctx.ResponseTransformers, gopher.ResponseTransformer{
Transformer: gopher.HeaderTransformerFor("all_gophermaps"),
Predicate: gopher.GopherMapPredicate })
```
In the code above, a simple Header is defined under the key "all\_gophermaps". There are no special keywords here; it can be anything. Then, a `ResponseTransformer` is added to the Context that applies the Headers to a Response (prepends the text to the content) found behind that same key, "all\_gophermaps", and reserve its use to gophermaps by tying it with the `Predicate` `GopherMapPredicate`. That way, text files will not come with this Header, only gophermaps.
## Executables
marmotte allows executables to be called and returns the result to the client. The executables can be of any nature: as long as their permissions in the filesystem allow the operating system user or group marmotte runs as to execute them, they represent valid resources.
The executables must return a single string, which should be either a path to a file, or a path to a directory. When generating data, the data should be placed in a file or in a collection of files. No data should be printed on `stdout` or `stderr`. It is generally a good idea to wrap an external program in a shell script and to make this script callable by marmotte. Files should be placed under the directory `$root/tmp`. The file `testdata/createtextfile.sh` is an example of such a shell script.
The executable should be referenced in a gophermap by giving the type of the file or files generated: for example, 0 for a text file, g for a graphics file, 1 for a directory (in case a collection of files are generated). For example:
```
0Generate a text file /createtextfile.sh mygopherhole.net 70
1Generate a collection of images /image-gen mygopherhole.net 70
```
## Error handling
marmotte offers dynamic error handling, similarly to what is found usually in http servers. Depending on the error encountered, a different message is reported to the user, with some information to help understand the error. As the Gopher Protocol leaves all the details of error handling to the server implementation, there are no standard error codes or specific mechanism expected by Gopher clients. marmotte handles error by redirecting a failed request to some content informing the user of the problem. This content is a template found in the `errors` subdirectory of `root`. There is one template per error code. Today, the templates are in plain text. marmotte defines the following error codes and templates:
|Error code|Description|Template|
|---|:---:|---|
|00|No error. This code is used when controlling the dispatching of the ResponseTransformers, to allow a pipeline to continue.|No template (normal response returned)|
|10|Unrecognised selector. This code is used whenever a selector has been submitted that, after going the RequestTransformer and ResponseTransformer pipelines hasn't been picked up by any ResponseTransformer.|`$root/$errors/10.gt`|
|20|Wrong permissions. This code is used whenever the server cannot access the resources behind a given selector.|`$root/$errors/20.gt`|
|30|Could not retrieve data. This error code is used whenever an error occur while reading the contents of a file. It is also used whenever an error template itself cannot be read.|`$root/$errors/30.gt`|
Error handling is done via a `ResponseTransformer` that comes with marmotte and that is set up by default.
### Customising error messages
marmotte comes with a set of default templates currently located in `testdata/errors/`. These files are Go templates and can be placed in `$root/$errors` and modified there. marmotte passes an instance of `GopherErrorInfo` to the templates. This data type is defined in the file `errorcodes.go`.
## Virtual folders and selector rewriting
marmotte offers selector rewriting facilities in the form of a RequestTransformer. The function `SelectorRewriteTransformerFor` takes a string `from` and a string `to` and returns a `RequestTransformerFunc` that replaces occurrences of `from` with `to` in the Path and FullPath fields of a Request instance. Both strings can be regular expressions, and `to` can reference matches found against `from` in the Request instance's FullPath via the usual `$1`, `$2`, etc.
## Users' gopherspace
marmotte offers the possibility for multiple users to have their own gopherspace. This is achieved through [selector rewriting](#virtual-folders-and-selector-rewriting). The individual gopherspaces are placed inside the directory `$root/$users_gopherspace`, and should be named after the users. They become available through tilde selectors, such as:
```
1Spock's gopher! ~spock enterprisesub.space 70
1Picard's phlog /~picard enterprisesub.space 70
```
The extra `/` in front of the selector is optional, and these two selectors will resolve as `$root/$users_gopherspace/spock` and `$root/$users_gopherspace/picard`, both typically directories, and as such will display as a gophermap.
## Limitations
marmotte is under active development, and currently does not implement all the features listed in the first section, [What is marmotte?](#what-is-marmotte).
@ -53,8 +132,13 @@ Below is a list of the features currently implemented. This is by no means a com
- [X] Gophermap Header Transformer
- [ ] Backwards-compatible selector/URI to offer and select Transformers
- [X] Request Transformers infrastructure
- [ ] Simple redirect based on Request Transformers
- [X] Simple redirect based on Request Transformers
- [X] Dynamic error content via templates
- [X] Dynamic content via templates
- [X] Dynamic execution of scripts and programs
- [ ] Image filtering via Transformers
## License
marmotte is published under the [2-Clause BSD License](https://tldrlegal.com/license/bsd-2-clause-license-(freebsd)). A [copy of the license](LICENSE) is included in the source repository.

@ -56,11 +56,11 @@ func (c *Context) AddRequestTransformerFor(s string, f RequestTransformerFunc) (
}
}
// DefaultResponseTransformers builds the default list of Request and Response Transformers for Context c. These Transformers are responsible for loading the data from the files.
// DefaultResponseTransformers builds the default list of Response Transformers for Context c. These Transformers are responsible for loading the data from the files.
func (c *Context) DefaultResponseTransformers() []ResponseTransformer {
executable := ResponseTransformer {
Transformer: ExecutableTransformer,
Predicate: FileExecutable }
Predicate: FileExecutablePredicate }
gophermap := ResponseTransformer{
Transformer: GopherMapTransformer,
Predicate: GopherMapPredicate }
@ -91,6 +91,7 @@ func (c *Context) DefaultResponseTransformers() []ResponseTransformer {
return c.ResponseTransformers
}
// DefaultRequestTransformers builds the default list of RequestTransformers for Context c. These Transformers are responsible for qualifying and re-qualifying the Request data.
func (c *Context) DefaultRequestTransformers() []RequestTransformer {
users_gopherspace := RequestTransformer{
Transformer: SelectorRewriteTransformerFor("^/?~(.+)", c.UsersGopherspace + "/$1"),

@ -67,11 +67,13 @@ func FileTypeDirectoryPredicate(c Context, rq Request, rs Response, e error) boo
return rq.Type == Directory
}
func FileExecutable(c Context, rq Request, rs Response, e error) bool {
// FileExecutablePredicate returns true if rq.FullPath is an accessible, is a file and is executable by the operating system user or group.
func FileExecutablePredicate(c Context, rq Request, rs Response, e error) bool {
fi, err := os.Stat(rq.FullPath)
return err == nil && fi.Mode()&0110 != 0
return err == nil && ! fi.IsDir() && fi.Mode()&0110 != 0
}
// GopherErrorPredicate returns true if rs.ErrorCode is not GopherErrorNoError or if both rs.ContentBinary and rs.ContentText are empty.
func GopherErrorPredicate(c Context, rq Request, rs Response, e error) bool {
return rs.ErrorCode != GopherErrorNoError || (len(rs.ContentBinary) == 0 && len(rs.ContentText) == 0)
}

@ -28,6 +28,7 @@ type Request struct {
Type RequestType
}
// GetRequestTypeFor returns the RequestType corresponding to the selector s. The types can be Directory, File or Unknown.
func GetRequestTypeFor(c Context, s string) RequestType {
fi, err := os.Stat(filepath.Clean(c.Root + "/" + s))
if err != nil {
@ -42,7 +43,6 @@ func GetRequestTypeFor(c Context, s string) RequestType {
return File
}
}
// NewRequest creates a new Request instance and returns a pointer to it. It relies on the selector s to determine the RequestType, and on the Context c to build the paths and verify the files.
func NewRequest(c Context, s string) (*Request, error) {
@ -62,34 +62,5 @@ func NewRequest(c Context, s string) (*Request, error) {
r.FullPath = c.Root + r.Path
r.Type = GetRequestTypeFor(c, s)
}
// } else if Selector.MatchString(s) {
// fi, err := os.Stat(filepath.Clean(c.Root + "/" + s))
// if err != nil {
// log.Error().
// Err(err).
// Msg("Received an error when trying to stat a full path")
// r.FullPath = c.Root + "/" + s
// r.Path = s
// r.Type = Unknown
// return &r, nil
// } else {
// r.Path = filepath.Clean("/" + s)
// r.FullPath = c.Root + r.Path
// if fi.IsDir() {
// r.Type = Map
// } else {
// r.Type = File
// }
// log.Debug().
// Str("Fullpath", r.FullPath).
// Send()
// }
// } else {
// // by default, consider that what is not a directory or a path to a file is 'NotImplemented'
// // later: this can be a function, etc.
// r.Type = NotImplemented
// return &r, fmt.Errorf("Unrecognised request: '%s'", s)
// }
return &r, nil
}

@ -11,6 +11,7 @@ import (
"github.com/rs/zerolog/log"
)
// write copies the contents of the Response rs onto the writer w. Depending on the ResponseType of rs, the data is written as text, with the sequence \r\n added after each line, and with a final .\r\n, or it is written as binary straight onto the writer.
func write(ctx *Context, rq Request, rs *Response, w io.Writer) error {
if rs.Type == PlainText {
for _, line := range rs.ContentText {
@ -69,6 +70,7 @@ func Transform(c *Context, rq Request, rs *Response, err error) []ResponseTransf
return transformers
}
// TransformRequest applies successively a list of RequestTransformers to a Request instance, tking into account the Context c. These Transformers are responsible for preparing the Request to be processed by the Responses. An example is selector rewriting. The Response parameter rs is currently not used.
func TransformRequest(c *Context, rq *Request, rs Response, err error) []RequestTransformer {
transformers := []RequestTransformer{}
ct, rqt, rst, et := c, rq, rs, err

@ -165,6 +165,7 @@ func ExecutableTransformer(c *Context, rq Request, rs *Response, err error) (*Co
return c, rq, rs, err
}
// ErrorRedirectTransformer is a ResponseTransformerFunc that substitutes error content to the ContentText of the Response rs. The error can come from earlier ResponseTransformer instances, and is then used as is to find the corresponding error template, or can be set to GopherSelectorNotFound in case both the ContentText and the ContentBinary are empty, indicating that no earlier ResponseTransformer instance picked up the Response. In case applying the error template itself encounters an issue, a default error content is placed in the Response instance. In all cases, the ResponseType is set to PlainText in the ResponseInstance.
func ErrorRedirectTransformer(c *Context, rq Request, rs *Response, err error) (*Context, Request, *Response, error) {
// If no content, no transformer could do anything: unknown selector
if len(rs.ContentBinary) == 0 && len(rs.ContentText) == 0 {
@ -188,7 +189,6 @@ func ErrorRedirectTransformer(c *Context, rq Request, rs *Response, err error) (
}
log.Error().
// Err(err).
Str("Error template path", p).
Msg("Error reading an error template")

Loading…
Cancel
Save