Proposal - Decreasing Reliance on Clay

5.18.23 | ~dachus-tiprel

This is a specific plan to implement the globular proposal, which in short, is to get rid of clay, and move it's functionality into either gall or userspace. Here is a brief list of reasons why:

• The mark system
  • prevents you from putting arbitrary data into your urbit
  • large parts of it go unused (e.g. grad, anything that isn't just noun<>json)
  • is confusing to lots of developers
  • can be replaced with something much simpler and more intuitive: a combination of jam, cue, ;;, and molds
• Is extremely complex
  • it is 5000 lines of code (most of which is not used)
  • large chunks of time have been spent trying to debug it's kernel machinery/app installation machinery
  • kernel upgrades could be much simpler
• Version control does not need to be in the kernel
  • "but then how will we do upgrades?" just make the code a solid state publication and push that out
  • It should be in userspace - see %linedb
• The build system doesn't (necessarily) have to be in the kernel either
  • Ford can live in gall, or in userspace
  • Moving it into userspace is interesting because it would allow people to write new build languages, instead of forcing everyone to the same set of build runes.

It is extremely difficult to simply "delete clay" because it is central to how the kernel/app upgrades are distributed and built. What we are going to do is build an alternative model alongside clay, and eventually make an atomic switchover to the new architecture. Once no one relies on clay anylonger, we can safely delete it.

This is the plan to break out all of Clay's functionality (file system, kernel/app distribution, version control, and build system) into simpler, separate, pieces.

Part 1 : Dill syncing

The first thing we will build is a new filesystem - one that lets you put arbitrary data into your urbit, publish it at a scry endpoint, and distribute it over the network. Baically: we want to give Urbit "Dropbox functionality" at the kernel level. Since we are just adding functionality to dill, not removing any functionality from clay, there should be no breaking changes, and this should be an easy first step.

Part 2 : New App Installation Architecture

Solid state publications can be built out of the new filesystem, which means we can start distributing source code. Since agents can read/write files into the namespace, we can build a bootloader agent that tells gall to e.g. "read from ~zod/%webterm and keep it up to date". Once the machinery is in place and functioning, we can do an atomic switchover from the old system (where clay tells gall what to run), to the new system (where gall tells itself what to run via a bootloader agent).

It is possible that we actually want a new userspace vane instead of modifying gall. It would probably look a lot more like shrubbery

Note that this is also additive, and doesn't mean that we have to eliminate Clay's installation architecture immediately. These can both co-exist side-by-side with no problems. At some point when we are comfortable, we can switch over all core upgrade logic machinery to use the new system, and it becomes very safe to delete clay.