Thursday, December 1, 2016
Ubuntu Core Gadget Snaps
Gagdet snaps, the somewhat mysterious part of snappy that few people grok. Being a distinct snap type, next to kernel, os and the most common app types, it gets some special roles. If you are on a classic system like Ubuntu, Debian or Fedora you don't really need or have one yet. Looking at all-snap core devices you will always see one. In fact, each snappy reference platform has one. But where are they?
Up until now the gadget snaps were a bit hard to find. They were out there but you had to have a good amount of luck and twist your tongue at the right angle to find them. That's all changed now. If you look a https://github.com/snapcore you will see a nice, familiar pattern of devicename-gadget. Each repository is dedicated to one device so you will see a gadget snap for Raspberry Pi 2 or Pi 3, for example.
But there's more! Each of those github repositories is linked to a launchpad project that automatically mirrors the git repository, builds the snap and uploads it to the store and publishes the snap to the edge channel!
The work isn't over, as you will see the gadget snaps are mostly in binary form, hand-made to work but still a bit too mysterious. The Canonical Foundations team is working on building them in a way that is friendlier to community and easier to trace back to their source code origins.
If you'd like to learn more about this topic then have a look at the snapd wiki page for gadget snaps.
Up until now the gadget snaps were a bit hard to find. They were out there but you had to have a good amount of luck and twist your tongue at the right angle to find them. That's all changed now. If you look a https://github.com/snapcore you will see a nice, familiar pattern of devicename-gadget. Each repository is dedicated to one device so you will see a gadget snap for Raspberry Pi 2 or Pi 3, for example.
But there's more! Each of those github repositories is linked to a launchpad project that automatically mirrors the git repository, builds the snap and uploads it to the store and publishes the snap to the edge channel!
The work isn't over, as you will see the gadget snaps are mostly in binary form, hand-made to work but still a bit too mysterious. The Canonical Foundations team is working on building them in a way that is friendlier to community and easier to trace back to their source code origins.
If you'd like to learn more about this topic then have a look at the snapd wiki page for gadget snaps.
Saturday, September 3, 2016
Gentoo overlay for snapd is now available
Thanks to a fantastic community contribution from Clayton "kefnab" Dobbs we now have a Gentoo overlay containing snapd [1].
I just wanted to shout a big thank you to mr Clayton. I'm very glad to see gentoo community interested in snapd and taking active participation in the project.
I will work on updating that to the latest version and as well as extending it to include the new snapd-xdg-open application.
[1] https://github.com/zyga/gentoo-snappy
I just wanted to shout a big thank you to mr Clayton. I'm very glad to see gentoo community interested in snapd and taking active participation in the project.
I will work on updating that to the latest version and as well as extending it to include the new snapd-xdg-open application.
[1] https://github.com/zyga/gentoo-snappy
Tuesday, August 9, 2016
Creating your first snappy interface
Today is a day I've been waiting for a long time. We now have enough knowledge to create our first real interface from scratch. To really understand this content you need to be familiar with parts [1], [2], [3] and [4].
We will go all the way, from branching snapd all the way to running a program that uses our new interface. We will focus on the ancillary tasks this time, the actual interface will be rather basic. Still, this knowledge will be invaluable next time where we will try to do something more complicated.
Adding the new "hello" interface
Let's get started. It all begins with snapd. If you didn't already, fork snapd and clone your fork locally. You may find this small guide that I wrote earlier useful. It goes through all those steps in detail. At the end of the exercise you should be able to build your fork of snapd (make sure it is really your fork, not the upstream version!)
Let's look around. Each time a new interface is added, the following files are modified:
- The file interfaces/builtin/foo{,_test}.go contains the actual interface
- The file interfaces/builtin/all{,_test}.go contains tiny change that is used to register a new interface
In addition, if the interface slot should implicitly show up on the core snap we need to change the file snap/implicit{,_test}.go. We will get back to this.
So let's create our new interface now. As mentioned in part [3] this entails implementing a new type with a few methods. Let's do that now:
Using your favorite code editor create a new file and save it as interfaces/builtin/hello.go. Paste the following code inside:
// -*- Mode: Go; indent-tabs-mode: t -*- /* * Copyright (C) 2016 Canonical Ltd * * This program is free software: you can redistribute it and/or modify * it under the terms of the GNU General Public License version 3 as * published by the Free Software Foundation. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program. If not, see <http://www.gnu.org/licenses/>. * */ package builtin import ( "fmt" "github.com/snapcore/snapd/interfaces" ) // HelloInterface is the hello interface for a tutorial. type HelloInterface struct{} // String returns the same value as Name(). func (iface *HelloInterface) Name() string { return "hello" } // SanitizeSlot checks and possibly modifies a slot. func (iface *HelloInterface) SanitizeSlot(slot *interfaces.Slot) error { if iface.Name() != slot.Interface { panic(fmt.Sprintf("slot is not of interface %q", iface)) } // NOTE: currently we don't check anything on the slot side. return nil } // SanitizePlug checks and possibly modifies a plug. func (iface *HelloInterface) SanitizePlug(plug *interfaces.Plug) error { if iface.Name() != plug.Interface { panic(fmt.Sprintf("plug is not of interface %q", iface)) } // NOTE: currently we don't check anything on the plug side. return nil } // ConnectedSlotSnippet returns security snippet specific to a given connection between the hello slot and some plug. func (iface *HelloInterface) ConnectedSlotSnippet(plug *interfaces.Plug, slot *interfaces.Slot, securitySystem interfaces.SecuritySystem) ([]byte, error) { switch securitySystem { case interfaces.SecurityAppArmor: return nil, nil case interfaces.SecuritySecComp: return nil, nil case interfaces.SecurityDBus: return nil, nil case interfaces.SecurityUDev: return nil, nil case interfaces.SecurityMount: return nil, nil default: return nil, interfaces.ErrUnknownSecurity } } // PermanentSlotSnippet returns security snippet permanently granted to hello slots. func (iface *HelloInterface) PermanentSlotSnippet(slot *interfaces.Slot, securitySystem interfaces.SecuritySystem) ([]byte, error) { switch securitySystem { case interfaces.SecurityAppArmor: return nil, nil case interfaces.SecuritySecComp: return nil, nil case interfaces.SecurityDBus: return nil, nil case interfaces.SecurityUDev: return nil, nil case interfaces.SecurityMount: return nil, nil default: return nil, interfaces.ErrUnknownSecurity } } // ConnectedPlugSnippet returns security snippet specific to a given connection between the hello plug and some slot. func (iface *HelloInterface) ConnectedPlugSnippet(plug *interfaces.Plug, slot *interfaces.Slot, securitySystem interfaces.SecuritySystem) ([]byte, error) { switch securitySystem { case interfaces.SecurityAppArmor: return nil, nil case interfaces.SecuritySecComp: return nil, nil case interfaces.SecurityDBus: return nil, nil case interfaces.SecurityUDev: return nil, nil case interfaces.SecurityMount: return nil, nil default: return nil, interfaces.ErrUnknownSecurity } } // PermanentPlugSnippet returns the configuration snippet required to use a hello interface. func (iface *HelloInterface) PermanentPlugSnippet(plug *interfaces.Plug, securitySystem interfaces.SecuritySystem) ([]byte, error) { switch securitySystem { case interfaces.SecurityAppArmor: return nil, nil case interfaces.SecuritySecComp: return nil, nil case interfaces.SecurityDBus: return nil, nil case interfaces.SecurityUDev: return nil, nil case interfaces.SecurityMount: return nil, nil default: return nil, interfaces.ErrUnknownSecurity } } // AutoConnect returns true if plugs and slots should be implicitly // auto-connected when an unambiguous connection candidate is available. // // This interface does not auto-connect. func (iface *HelloInterface) AutoConnect() bool { return false }
TIP: Any time you are making code changes use go fmt to re-format all of the code in the current working directory to the go formatting standards. Static analysis checkers in the snappy tree enforce this so your code won't be able to land without first being formatted correctly.
This code is perfectly fine, if a little verbose (we'll get it shorter eventually, I promise). Now let's make one more change to ensure the interface known to snapd. All we need to do is to add it to the allInterfaces list in the same golang package. I've decided to just use an init() function so that all of the changes are in one file and cause less conflicts for other developers creating their interfaces. You can see my patch below.
diff --git a/interfaces/builtin/all_test.go b/interfaces/builtin/all_test.go index 46ca587..86c8fad 100644 --- a/interfaces/builtin/all_test.go +++ b/interfaces/builtin/all_test.go @@ -62,4 +62,5 @@ func (s *AllSuite) TestInterfaces(c *C) { c.Check(all, DeepContains, builtin.NewCupsControlInterface()) c.Check(all, DeepContains, builtin.NewOpticalDriveInterface()) c.Check(all, DeepContains, builtin.NewCameraInterface()) + c.Check(all, Contains, &builtin.HelloInterface{}) } diff --git a/interfaces/builtin/hello.go b/interfaces/builtin/hello.go index d791fc5..616985e 100644 --- a/interfaces/builtin/hello.go +++ b/interfaces/builtin/hello.go @@ -130,3 +130,7 @@ func (iface *HelloInterface) PermanentPlugSnippet(plug *interfaces.Plug, securit func (iface *HelloInterface) AutoConnect() bool { return false } + +func init() { + allInterfaces = append(allInterfaces, &HelloInterface{}) +}
Now switch to the snap/ directory, edit the file implicit.go and add "hello" to implicitClassicSlots. You can see the whole change below. I also updated the test that checks the number of implicitly added slots. This change will make snapd create a hello slot on the core snap automatically. As you can see by looking at the file, there are many interfaces that take advantage of this little trick.
diff --git a/snap/implicit.go b/snap/implicit.go index 3df6810..098b312 100644 --- a/snap/implicit.go +++ b/snap/implicit.go @@ -60,6 +60,7 @@ var implicitClassicSlots = []string{ "pulseaudio", "unity7", "x11", + "hello", } // AddImplicitSlots adds implicitly defined slots to a given snap. diff --git a/snap/implicit_test.go b/snap/implicit_test.go index e9c4b07..364a6ef 100644 --- a/snap/implicit_test.go +++ b/snap/implicit_test.go @@ -56,7 +56,7 @@ func (s *InfoSnapYamlTestSuite) TestAddImplicitSlotsOnClassic(c *C) { c.Assert(info.Slots["unity7"].Interface, Equals, "unity7") c.Assert(info.Slots["unity7"].Name, Equals, "unity7") c.Assert(info.Slots["unity7"].Snap, Equals, info) - c.Assert(info.Slots, HasLen, 29) + c.Assert(info.Slots, HasLen, 30) } func (s *InfoSnapYamlTestSuite) TestImplicitSlotsAreRealInterfaces(c *C) {
Now let's see what we have. You should have three patches:
- The first one adding the dummy hello interface
- The second one registering it with allInterfaces
- The third one adding it to implicit slots on the core snap, on classic
This is how it looked like for me. You can also see the particular commit message style I was using. All the commits are prefixed with the directory where the changes were made, followed by a colon and by the summary of the change. Normally I would also add longer descriptions but here this is not required as the changes are trivial.

Seeing our interface for the first time
Let's run our changed code with
devtools. Switch to the devtools directory and run:./refresh-bits snapd setup run-snapd restore
This roughly means:
- Build snapd from source (based on correctly set $GOPATH)
- Prepare for running locally built snapd
- Run locally built snapd
- Restore regular version of snapd
The script will prompt you for your password. This is required to perform the system-wide operations. It will also block and wait until you interrupt it by pressing control-c. Please don't do this yet though, we want to check if our changes really made it through.

To check this we can simply list the available interfaces with
sudo snap interfaces | grep hello
Why sudo? Because of the particular peculiarity of how refresh-bits works. Normally it is not required. Did it work? It did for me:

TIP: If it didn't work for you and you didn't get the hello interface then the most likely cause of the issue is that you were editing your own fork but refresh-bits still built the vanilla upstream version that is checked out somewhere else.
Go to $GOPATH/src/github.com/snapcore/snapd and ensure that this is indeed the fork you were expecting. If not just remove this directory and move your fork (that you may have cloned elsewhere) here and try again.
Great. Now we are in business. Let's recap what we did so far:
- We added a whole new interface by dropping boilerplate code into interfaces/builtin/hello.go
- We registered the interface in the list of allInterfaces
- We made snapd inject an implicit (internally defined) slot on the core snap when running on classic
- We used refresh-bits to run our locally built version and confirmed it really works
Granting permissions through interfaces
With the scaffolding in place we can now work on using our new interface to actually grant additional permissions. When designing real interfaces you have to think about what kind of action should grant extra permissions. Recall from part [3] that you can freely associate extra security snippets that encapsulate permissions either permanent or connection-based snippets on both plugs and slots.
A permanent snippet is always there as long as a plug or a slot exists. Typically this is used on snaps other than the core snap (e.g. a service provider that is packaged as a snap) to allow the service to run. Examples of such services could include network-manager, modem-manager, docker, lxd, x11 any anything like that. The granted permissions allow the service to operate as well as to create a communication channel (typically a socket or a DBus bus name) that clients can connect to.
Connection based snippet is only applied when a connection (interface connection) is made. This permission is given to the client and typically involves basic IPC system calls as well as a permission to open a given socket (e.g. the X11 socket) or use specific DBus methods and objects.
A special class of connection snippets exist for things that don't involve a client application talking to a server application but rather a program using given kernel services. The prime example of this use case is the network interface which grants access to several network system calls. This interface grants nothing to the slot, just to connected plugs.
Today we will explore that last case. Our hello interface will give us access to a system call that is usually forbidden, reboot
The graceful-reboot snap
For the purpose of this exercise I wrote a tiny C program that uses the reboot() function. The whole program is reproduced below, along with appropriate build support for snapcraft.
graceful-reboot.c
#include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <sys/reboot.h> #include <linux reboot.h> #include <errno.h> int main() { sync(); if (reboot(LINUX_REBOOT_CMD_RESTART) != 0) { switch (errno) { case EPERM: printf("Insufficient permissions to reboot the system\n"); break; default: perror("reboot()"); break; } return EXIT_FAILURE; } printf("Reboot requested\n"); return EXIT_SUCCESS; }
Makefile
TIP: Makefiles rely on differences between tabs and spaces. When copy pasting this sample you need to ensure that tabs are preserved in the clean and install rules
CFLAGS += -Wall .PHONY: all all: graceful-reboot .PHONY: clean clean: rm -f graceful-reboot graceful-reboot: graceful-reboot.c .PHONY: install install: graceful-reboot install -d $(DESTDIR)/usr/bin install -m 0755 graceful-reboot $(DESTDIR)/usr/bin/graceful-reboot
snapcraft.yaml
name: graceful-reboot version: 1 summary: Reboots the system gracefully description: | This snap contains a graceful-reboot application that requests the system to reboot by talking to the init daemon. The application uses a custom "hello" interface that is developed as a part of a tutorial. confinement: strict apps: graceful-reboot: command: graceful-reboot plugs: [hello] parts: main: plugin: make source: .
We now have everything required. Now let's use snapcraft to build this code and get our snap ready. I'd like to point out that we use confinement: strict. This tells snapcraft and snapd that we really really don't want to run this snap in devmode where all sandboxing is off. Doing so would defeat the purpose of adding the new interface.
Let's install the snap and try it out:
$ snapcraft $ sudo snap install ./graceful-reboot_1_amd64.snap
Ready? Let's run the command now (don't worry, it won't reboot)
$ graceful-reboot Bad system call
Aha! That's what we are here to fix. As we know from part [4] that seccomp, which is a part of the sandbox system, blocks all system calls that are not explicitly allowed. We can check the system log to see what the actual error message was to figure out which system call need to be added.
Looking through journalctl -n 100 (last 100 messages) I found this:
sie 10 09:47:02 x200t audit[13864]: SECCOMP auid=1000 uid=1000 gid=1000 ses=2 pid=13864 comm="graceful-reboot" exe="/snap/graceful-reboot/x1/usr/bin/graceful-reboot" sig=31 arch=c000003e syscall=169 compat=0 ip=0x7f7ef30dcfd6 code=0x0
Decoding this is rather cryptic message is actually pretty easy. The key fact we are after is the system call number. Here it is 169. Which symbolic system call is that? We can use the scmp_sys_resolver program to find out.
$ scmp_sys_resolver 169 reboot
Bingo. While this case was pretty obvious, library functions don't always map to system call names directly. There are often cases where system calls evolve to gain additional arguments, flags and what not and the name of the library function is not changed.
Adjusting the hello^Hreboot interface
At this stage we can iterate on our interface. We have a test case (the snap we just wrote), we have the means to change the code (editing the hello.go file) and to make it live in the system (the refresh-bits command).
Let's patch our interface to be a little bit more meaningful now. We will rename it from hello to reboot. This will affect the file name, the type name and the string returned from Name(). We will also grant a connected plug permission, through the seccomp security backend, to the use the reboot system call. You can try to make the necessary changes yourself but I provide the essential part of the patch below. The key thing to keep in mind is that ConnectedPlugSnippet method must return, when asked about SecuritySecComp, the name of the system call to allow, like this []byte(`reboot`).
diff --git a/interfaces/builtin/reboot.go b/interfaces/builtin/reboot.go index 91962e1..ba7a9e3 100644 --- a/interfaces/builtin/reboot.go +++ b/interfaces/builtin/reboot.go @@ -91,7 +91,7 @@ func (iface *RebootInterface) PermanentSlotSnippet(slot *interfaces.Slot, securi func (iface *RebootInterface) ConnectedPlugSnippet(plug *interfaces.Plug, slot *interfaces.Slot, securitySystem interfaces.SecuritySystem) ([]byte, error) { switch securitySystem { case interfaces.SecurityAppArmor: return nil, nil case interfaces.SecuritySecComp:
- return nil, nil+ return []byte(`reboot`), nilcase interfaces.SecurityDBus:
We can now return to the terminal that had refresh-bits running, interrupt the running command with control-c and simply re-start the whole command again. This will build the updated copy of snapd.
Surely enough, if we now ask snap about known interfaces we will see the reboot interface.
Let's now update the graceful-reboot snap to talk about the reboot interface and re-install it (note that you don't have to change the version string at all, just re-install the rebuilt snap). If we ask snapd about interfaces now we will see that the core snap exposes the reboot slot and the graceful-reboot snap has a reboot plug but they are disconnected.

Let's connect them now.

Let's connect them now.
sudo snap connect graceful-reboot:reboot ubuntu-core:reboot
Let's ensure that it worked by running the interfaces command again:
$ sudo snap interfaces | grep reboot :reboot graceful-reboot
Success!
Now before we give it a try, let's have a look at the generated seccomp profile. The profile is derived from the base seccomp template that is a part of snapd source code and the set of plugs, slots and connections made to or from this snap. Each application of each snap gets a profile, we can see that for ourselves by looking at
/var/lib/snapd/seccomp/profiles/snap.graceful-reboot.graceful-reboot
/var/lib/snapd/seccomp/profiles/snap.graceful-reboot.graceful-reboot
$ grep reboot /var/lib/snapd/seccomp/profiles/snap.graceful-reboot.graceful-reboot reboot
There are a few gotchas that we should point out though:
- Security profiles are derived from interfaces but are only changed when a new connection is made (and that connection affects a particular snap), when the snap is initially installed or every time it is updated.
- In practice we will either disconnect / reconnect the hello interface or reinstall the snap (whichever is more convenient)
- Snapd remembers connections that were made explicitly and will re-establish them across snap updates. If you rename an interface while working on it, snapd may print a message (to system log, not to the console) about being unable to reconnect the "hello" interface because that interface no longer exists in snapd. To make snapd forget all those connections simply remove and reinstall the affected snap.
- You can experiment by editing seccomp profiles directly. Just edit the file mentioned above and add additional system calls. Once you are happy with the result you can adjust snapd source code to match.
- You can also do that with apparmor profiles but you have to re-load the profile into the kernel each time using the command apparmor_parser -r /path/to/the/profile
Okay, so did it work? Let's try the graceful-reboot command again.
$ graceful-reboot Insufficient permissions to reboot the system
The process didn't get killed straight away but the reboot call didn't work yet either. Let's look at the system log and see if there are any hints.
sie 10 10:25:55 x200t kernel: audit: type=1400 audit(1470817555.936:47): apparmor="DENIED" operation="capable" profile="snap.graceful-reboot.graceful-reboot" pid=14867 comm="graceful-reboot" capability=22 capname="sys_boot"
This time we could use the system call but apparmor stepped in and said that we need the sys_boot capability. Let's modify the interface to allow that then. The precise apparmor syntax for this is "capability sys_boot,". You absolutely have to include the trailing comma. It was my most common mistake when hacking on apparmor profiles.
Let's patch the interface and re-run refresh-bits and re-install the snap. You can see the patch I applied below
diff --git a/interfaces/builtin/reboot.go b/interfaces/builtin/reboot.go index 91962e1..ba7a9e3 100644 --- a/interfaces/builtin/reboot.go +++ b/interfaces/builtin/reboot.go @@ -91,7 +91,7 @@ func (iface *RebootInterface) PermanentSlotSnippet(slot *interfaces.Slot, securi func (iface *RebootInterface) ConnectedPlugSnippet(plug *interfaces.Plug, slot *interfaces.Slot, securitySystem interfaces.SecuritySystem) ([]byte, error) { switch securitySystem { case interfaces.SecurityAppArmor: - return nil, nil + return []byte(`capability sys_boot,`), nil case interfaces.SecuritySecComp: return []byte(`reboot`), nil case interfaces.SecurityDBus:
TIP: I'm using sudo with a full path because of a bug in sudo where /snap/bin is not kept on the path.
Warning: the next command will reboot your system straight away!
$ sudo /snap/bin/graceful-reboot
Final touches
Since interfaces like this are very common, snapd has some support for writing less repetitive code. The exactly identical interface can be written with just this short snippet
// NewRebootInterface returns a new "reboot" interface. func NewRebootInterface() interfaces.Interface { return &commonInterface{ name: "reboot", connectedPlugAppArmor: `capability sys_boot,`, connectedPlugSecComp: `reboot`, reservedForOS: true, } }
Everywhere where we used to say &RebootInterface{} we can now say NewRebootInterface(). The abbreviation helps in code reviews and in just conveying the meaning and intent of the interface.
You can find the source code of this interface in my github repository. The code of the graceful reboot snap is here. Feel free to comment below, or on Google+ or ask me questions directly.
Monday, August 8, 2016
Snap execution environment
This is the fourth article in the series about snappy interfaces. You can check out articles one, two and three though they are not directly required.
In this installment we will explore the layout and properties of the file system at the time snap application is executed. From the point of view of the user nothing special is happening. The user either runs an application by clicking on a desktop icon or by running a shell command. Internally, snapd uses a series of steps (which I will not explain today as they are largely an implementation detail) to configure the application process.
The (ch)root filesystem and a bit of magic
Let’s start with the most important fact: the root filesystem is not the filesystem of the host distribution. Using the host filesystem would lead to lots of inconsistencies. Those are rather obvious: different base libraries, potentially different filesystem layout, etc. At snap application runtime the root filesystem is changed to the core snap itself. You can think of this as a kind of chroot. Obviously the chroot itself would be insufficient as snaps are read only filesystems and the core snap is no different.
Certain directories in the core snap are bind mounted (you can think of this as a special type of a symbolic link or a hard link to a directory though neither are fully accurate) to locations on the host file system. This includes the /home directory, the /run directory and a few others (see Appendix A for the full list). Notably this does not include the /usr/lib or /usr/bin. If a snap needs a library or an executable to function, that library or executable has to be present in the snap itself. The only exception to that are very low level libraries like libc that are present in the core snap.
TIP: explore the core snap to see what is there. Having installed at least one snap you can go to /snap/ubuntu-core/current to see the list of files provided there.
With all those mounts and chroots in place one might wonder how mounts look like for all the other processes in the system? The answer is simple, they look as if nothing special related to snappy was happening.
To understand the answer you have to know a little about Linux namespaces. Namespaces are a way to create a separate view of a given aspect of a Linux system at a per-process level. Unlike full-blown virtual machines (where you run a whole emulated computer with a potentially different operating system kernel and emulated peripheral devices) namespaces are fine grained. Snappy uses just one of the available namespaces, the mount namespace. Now I won’t fool you, while the idea seems simple “mounts in the namespace are isolated from the mounts outside of the namespace” the reality is far more complex because of the so-called shared-subtrees. One interesting consequence is that mounts performed after a snap application is stated (e.g. in the /media directory) are visible to the said application (e.g. to VLC) while the reverse is not true. If a malicious snap tries (and manages despite various defenses put in place) to mount something in say, /usr/ that change will be visible only to the snap application process.
Don’t worry if you don’t fully understand this topic. The main point is that your application sees a different view of the filesystem but that view is consistent across distributions.
TIP: you may have seen the core snap as it looks like on disk if you followed the earlier tip. Now see the real file system at runtime! Install the snapd-hacker-toolbelt snap and run snapd-hacker-toolbelt.busybox sh. This will give you a shell with many of the familiar commands that let you peek and poke at the environment.Now there are a few more tweaks I should point out but won’t go into too much detail:
- Each process gets a private /tmp directory with a fresh tmpfs mounted there. This is a security measure. One simple consequence is that you cannot expect to share files by dropping them there and that you cannot create arbitrarily large files there since tmpfs is backed by a fraction of available system memory.
- There’s also a private instance of /dev/pts directory with pseudo terminal emulators. This is an another security measure. In practice you will not care about this much. It’s just a part of the Linux plumbing that has to be setup in a given way.
- The whole host filesystem is mounted at /var/lib/snapd/hostfs. This can be used by interfaces similar to the content sharing interface, for example. This is super interesting and we will devote a whole article to using this later on.
- There’s special code that exists to support Nvidia proprietary drivers. I will discuss this with a separate installment that may be of interest to game developers.
- The current working directory may be impossible to preserve across the whole chroot and mount and bind mount magic. The easiest way to experience this is to create a directory in /tmp (e.g. /tmp/foo) and try to run any snap command there. Because of the private (and empty) /tmp directory the /tmp/foo directory does not exist for the snap application process. Snap-confine will print an informative error message and refuse to run.
This now much more comfortable. Many of the usual places exist and contain the data the applications are familiar with. This doesn’t mean those directories are readable or writable to the application process, they are just present. Confinement and interfaces decide if something is readable or writable. This brings us to the second big part of snap-confine
Process confinement
Snap-confine (as of version 1.0.39) supports two sandboxing technologies: seccomp and apparmor.Seccomp is used to constrain system calls that an application can use. System calls are the interface between the kernel and user space. If you are unfamiliar with the concept then don’t worry. In very rough terms some of the functions of your programming language are implemented as as system calls and seccomp is the linux subsystem that is responsible for mediating access to them.
Right now when your application runs in devmode you will not get any advice on the system calls you are relying on that are not allowed by the set of used interfaces. The so-called complain mode of seccomp is being actively developed upstream so the situation may change by the time you are reading this.
In strict confinement any attempt to use a disallowed system call will instantly kill the offending process. This is accompanied by a rather cryptic message that you can see in the system log:
sie 08 12:36:53 gateway kernel: audit: type=1326 audit(1470652613.076:27): auid=1000 uid=1000 gid=1000 ses=63 pid=66834 comm="links" exe="/snap/links/2/usr/bin/links" sig=31 arch=c000003e syscall=54 compat=0 ip=0x7f8dcb8ffc8a code=0x0What this tells us is that process ID 66834 was killed with signal 31 (SIGSYS) because it tried to use system call 54 (setsockopt) Note that system call numbers are architecture specific. The output above was from an amd64 machine.
Even when a system call is allowed the particular operation may be intercepted and denied by apparmor. For example, the sandbox is setup so that applications can freely write to $SNAP_USER_DATA (or $SNAP_DATA for services) but cannot, by default, either read or write from the real home directory.
sie 08 12:56:40 gateway kernel: audit: type=1400 audit(1470653800.724:28): apparmor="DENIED" operation="open" profile="snap.snapd-hacker-toolbelt.busybox" name="/home/zyga/.ssh/authorized_keys" pid=67013 comm="busybox" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Here we see that process ID 67013 tried to “open” /home/zyga/.ssh/authorized_keys and that the “r” (read) mask was denied. In devmode that is obviously allowed but is accompanied with an appropriate warning message instead.
TIP: Whenever you run into problems like this you should give the snappy-debug snap a try. It is designed to read and understand messages like that and give you useful advice.
Apparmor has much wider feature set and can perform checks for linux capabilities, traditional UNIX IPC like signals and sockets, DBus messages (including details of the object and method invoked). The vast majority of the current snap confinement is made with apparmor profiles. We will look at all the features in greater detail in the next few articles in this series where we will actively implement simple new interfaces from scratch.
There's one last thing that snap-confine does, in some cases is creates...
A device control group
In certain cases a device control group is created and the application process is moved there. The control group has only a small set of typical device nodes (e.g. /dev/null) and explicitly defined additional devices. This is done by using udev rules to tag appropriate devices. This is listed for completeness, you should never worry about or even think about this topic. Once the need arises we will explore it and how it can be used to simplify handling of certain situations. Again, by default there is no device control group being used.
Putting it all together
With all of those changes in place snap-confine executes (using execv) a wrapper script corresponding to the application command entry in the snapcraft.yaml file. This happens each time you run an application.If you are interested in learning more about snap-confine I encourage you to check out its manual page (snap-confine.5) and source code. If you have any questions please feel free to ask at the snapcraft.io mailing list or using comments on this blog below.
Next time we will look at creating our first, extremely simple, interface in practice.
Appendix A: List of host directories that are bind-mounted.
NOTE: This list will slowly get less and less accurate as more of the mount points become dynamic and controlled by available interfaces.- /dev
- /etc (except for /etc/alternatives)
- /home
- /root
- /proc
- /snap
- /sys
- /var/snap
- /var/lib/snapd
- /var/tmp
- /var/log
- /run
- /media
- /lib/modules
- /usr/src
Tuesday, July 5, 2016
Fresh bite-sized bugs in snappy
Do you remember my previous post about bite-sized bugs in snappy? Those are bugs that are small and trivial to fix. A perfect way to get involved in a project or to get your feet wet in a new programming language.
I've just noticed that there are plenty of new bugs that have been tagged "bitesized" in Snapcraft. As before I'd like to extend my invitation to the the community to have a look and send patches, pull requests or just a ping if they are interested in fixing them.
It usually takes just a few moments to understand the bug and a few moments to come up with a fix locally. While you do that you'll learn a lot about how the project works and, perhaps, you will make your first public contribution to a free software project (I remember how the journey started for me years ago :-)
Please give it a try, you can always ping us in #snappy on Freenode with any questions.
I've just noticed that there are plenty of new bugs that have been tagged "bitesized" in Snapcraft. As before I'd like to extend my invitation to the the community to have a look and send patches, pull requests or just a ping if they are interested in fixing them.
It usually takes just a few moments to understand the bug and a few moments to come up with a fix locally. While you do that you'll learn a lot about how the project works and, perhaps, you will make your first public contribution to a free software project (I remember how the journey started for me years ago :-)
Please give it a try, you can always ping us in #snappy on Freenode with any questions.
Monday, July 4, 2016
snapd 2.0.10 released to Fedora COPR
Hey there snappers
Fedora users can now get snapd 2.0.10 from the COPR repository. There are many bug fixes and new features in this release.
- interfaces: also allow @{PROC}/@{pid}/mountinfo and
- @{PROC}/@{pid}/mountstats
- interfaces: allow read access to /etc/machine-id and
- @{PROC}/@{pid}/smaps
- interfaces: miscelleneous policy updates for default, log-observe and system-observe
- snapstate: add logging after a successful doLinkSnap
- tests, integration-tests: port try tests to spread
- store, cmd/snapd: send a basic user-agent to the store
- store: add buy method
- client: retry on failed GETs
- tests: actual refresh test
- docs: REST API update
- interfaces: add mount support for hooks.
- interfaces: add udev support for hooks.
- interfaces: add dbus support for hooks.
- tests, integration-tests: port refresh test to spread
- tests, integration-tests: port change errors test to spread
- overlord/ifacestate: don't retry snap security setup
- integration-tests: remove unused file
- tests: manage the socket unit when reseting state
- overlord: improve organization of state patches
- tests: wait for snapd listening after reset
- interfaces/builtin: allow other sr*/scd* optical devices
- systemd: add support for squashfuse
- snap: make snaps vanishing less fatal for the system
- snap-exec: os.Exec() needs argv0 in the args[] slice too
- many: add new `create-user` command
- interfaces: auto-connect content interfaces with the same content and developer
- snapstate: add Current revision to SnapState
- readme: tweak readme blurb
- integration-tests: wait for listening port instead of active
- service reported by systemd
- many: rename Current -> {CurrentSideInfo,CurrentInfo}
- spread: fix home interface test after suite move
- many: name unversioned data.
- interfaces: add "content" interface
- overlord/snapstate: defaultBackend can go away now
- debian: comment to remember why the timer is setup like it is
- tests,spread.yaml: introduce an upgrade test, support/split into two suites for this
- overlord,overlord/snapstate: ensure we keep snap type in snapstate of each snap
- many: rework the firstboot support
- integration-tests: fix test failure
- spread: keep core on suite restore
- tests: temporary fix for state reset
- overlord: add infrastructure for simple state format/content migrations
- interfaces: add seccomp support for hooks.
- interfaces: allow gvfs shares in home and temporarily allow
- socketcall by default (LP: #1592901, LP: #1594675)
- tests, integration-tests: port network-bind interface tests to spread
- snap,snap/snaptest: use PopulateDir/MakeTestSnapWithFiles directly and remove MockSnapWithHooks
- interfaces: add mpris interface
- tests: enable `snap run` on i386
- tests, integration-tests: port network interface test to spread
- tests, integration-tests: port interfaces cli to spread
- tests, integration-tests: port leftover install tests to spread
- interfaces: add apparmor support for hooks.
- tests, integration-tests: port log-observe interface tests to spread
- asserts: improve Decode doc comment about assertion format
- tests: moved snaps to lib
- many: add the camera interface
- many: add optical-drive interface
- interfaces: auto-connect home if running on classic
- spread: bump gccgo test timeout
- interfaces: use security tags to index security snippets.
- daemon, overlord/snapstate, store: send confinement header to the store for install
- spread: run tests on 16.04 i386 concurrently
- tests,integration-tests: port install error tests to spread
- interfaces: add a serial-port interface
- tests, integration-tests, debian: port sideload install tests to spread
- interfaces: add new bind security backend and refactor backendtests
- snap: load and validate implicit hooks.
- tests: add a build/run test for gccgo in spread
- cmd/snap/cmd_login: Adjust message after adding support for wheel group
- tests, integration-tests: ported install from store tests ton spread
- snap: make `snap change <taskid>` show task progress
- tests, integration-tests: port search tests to spread
- overlord/state,daemon: make abort proceed immediately, fix doc comment, improve tests
- daemon: extend privileged access to users in "wheel" group
- snap: tweak `snap refresh` and `snap refresh --list` outputTiny
- interfaces: refactor auto-connection candidate check
- snap: add support for snap {install,refresh} --{edge,beta,candidate,stable}
- release: don't force KDE Neon into devmode.
Snappy in Arch moved to community repo
Hey there snappers!
I’d like to announce something that you may have noticed during the last update of snapd to version 2.0.10. The AUR package is no longer there, instead you can now get and update snappy on Arch simply by running this one-liner:
„pacman -S snapd"
That’s right, snapd and snap-confine have now moved to the official community repository. This means that the barrier to entry is now significantly lower and that installation is even faster than before. You still want to read the snapd wiki page to know the details about various post-install activities.
Tuesday, June 28, 2016
The /etc/os-release zoo
If you've ever wanted to do something differently depending on the /etc/os-release but weren't in the mood of installing every common distribution under the sun, look no further. I give you the /etc/os-release zoo project.
A project like this is never complete so please feel free to contribute additional distribution bits there.
A project like this is never complete so please feel free to contribute additional distribution bits there.
Wednesday, June 1, 2016
Bug #1587445 is no more; Thank you Thibran!
I just wanted to quickly give public thanks to Thibran who quickly fixed bug 1587445. This bug was opened less than 24 hours ago. Thibran forked the code, followed our hacking instructions, proposed the fix on github, talked to us on IRC (#snappy) and got his contribution merged.
Thank you Thibran
This is how free software is made!
Making your first contribution to snapd
Making your first contribution to snapd is quite easy but if you are not familiar with how go projects are structured it may be non-obvious where the code is or why your forked branch is not being used. Let's see how to do this step by step:
- Fork the project on github. You will need a github account.
- Set the GOPATH environment variable to something like ~/hacking or ~/fun or like I do ~/work (though I must say my work is full of fun hacking :-). You can set it directly with export GOPATH=~/hacking but I would recommend to add it to your ~/.profile file. Note that it will only be used on the next login so you can also set it directly or source the ~/.profile file with . ~/.profile after you've made the change.
- Set PATH to include $GOPATH/bin -- you will need this to run some of the programs you build and install (export PATH=$PATH:$GOPATH/bin) -- also make this persistent as described above
- Create $GOPATH/src/snapcore (mkdir -p $GOPATH/src/github.com/snapcore) and enter that directory (cd $GOPATH/src/github.com/snapcore)
- Clone your fork here (git clone git@github.com:zyga/snapd -- do replace zyga with your github account name)
- Add the upstream repository as a remote (git remote add upstream git@github.com:snapcore/snapd) - you will use this later to get latest changes from everyone
- Run the ./get-deps.sh script. You will probably need a few extra things (sudo apt-get install golang-go bzr)
- You are now good to go :-)
At this stage you can go build and go test individual packages. I would also advise you to have a look at my devtools repository where I've prepared useful scripts for various tasks. One thing you may want to use is the refresh-bits script. Having followed steps 1 through 8 above, let's use refresh-bits to run our locally built version of snapd and snap.
- Clone devtools to anywhere you want (git clone git@github.com:zyga/devtools) and enter the cloned repository (cd devtools)
- Assuming your GOPATH is still set, run ./refresh-bits snap snapd setup run-snapd restore
- Look at what is printed on screen, at some point you will notice that locally started snapd is running (you can always stop it by pressing ctrl+c) along with instructions on how to run your locally built snap executable to talk to it
- In another terminal, in the same directory, run sudo ./snap.amd64 --version
Voila :-)
If you have any questions about that, please use the comments section below. Thanks!
Tuesday, May 31, 2016
Bite-size bugs in snapd
Hey
This is just a quick shout to anyone out there that is interested in snapd, maybe follows development or is just curious. We are getting more and more bite-size bugs that would be perfect for someone to pick up and fix as their first contribution to either free and open source software in general or perhaps to specifically to snapd itself.
There's one such bug I'd like to highlight just now. https://bugs.launchpad.net/snappy/+bug/1587445
The bug is very simple to fix. The snap list command should say something appropriate when there are no snaps installed. This is literally an if statement and a printf call. If anyone wants to try to do that, feel free to ping me on irc (zyga on #snappy on freenode) or comment below on Google+.
We have hacking instructions, we have automated tests and we have a friendly development community. I'd love to see you joint us.
This is just a quick shout to anyone out there that is interested in snapd, maybe follows development or is just curious. We are getting more and more bite-size bugs that would be perfect for someone to pick up and fix as their first contribution to either free and open source software in general or perhaps to specifically to snapd itself.
There's one such bug I'd like to highlight just now. https://bugs.launchpad.net/snappy/+bug/1587445
The bug is very simple to fix. The snap list command should say something appropriate when there are no snaps installed. This is literally an if statement and a printf call. If anyone wants to try to do that, feel free to ping me on irc (zyga on #snappy on freenode) or comment below on Google+.
We have hacking instructions, we have automated tests and we have a friendly development community. I'd love to see you joint us.
snapd 2.0.5 released, new release cadence
There's a new release of snapd arriving in Ubuntu 16.04. As before, our fearless release manager Michael Vogt has crafted the work and made sure it can arrive to your machines on a timely basis.
You can see the changelog below, annotated with links to fixed bugs. I would only like to highlight one bug which improves experience of snaps under Unity 7.
New snapd releases are now planned to happen every week. You can expect a steady stream of fresh snappy goodness in both snapd and in the store. With this in mind we also plan to change the version scheme. Currently, as you can see below, we use 2.0.x for each micro-release. This system will quickly get meaningless so we will likely witch to a date-based release names instead . Expect to see a 2016W22 (or ..23) release next time around.
On the development front, many interesting changes are in the pipeline. While not a part of snapd 2.0.5 they should be released in the next few weeks, at most. You can expect applications to gain ability to play sound and music using the new pulseaudio interface. This ability, along with bug fixes to opengl should unlock the ability to deliver many popular games as snaps. Game on!
There's also ongoing work to allow sharing data from the classic Ubuntu and snaps. One of our first goals is to allow sharing fonts. This will improve the user experience of snaps that want to take advantage of custom, locally installed fonts. It should also allow us to package fonts as snaps in the near future. The underlying technology is very generic and I'm sure we'll find many interesting things to share this way.
As always, you can reach out to us on IRC (#snappy) and on the mailing list (snapcraft@lists.ubuntu.com, see this post for details). If you have any questions we will be happy to answer them.
See you next week!
You can see the changelog below, annotated with links to fixed bugs. I would only like to highlight one bug which improves experience of snaps under Unity 7.
New snapd releases are now planned to happen every week. You can expect a steady stream of fresh snappy goodness in both snapd and in the store. With this in mind we also plan to change the version scheme. Currently, as you can see below, we use 2.0.x for each micro-release. This system will quickly get meaningless so we will likely witch to a date-based release names instead . Expect to see a 2016W22 (or ..23) release next time around.
On the development front, many interesting changes are in the pipeline. While not a part of snapd 2.0.5 they should be released in the next few weeks, at most. You can expect applications to gain ability to play sound and music using the new pulseaudio interface. This ability, along with bug fixes to opengl should unlock the ability to deliver many popular games as snaps. Game on!
There's also ongoing work to allow sharing data from the classic Ubuntu and snaps. One of our first goals is to allow sharing fonts. This will improve the user experience of snaps that want to take advantage of custom, locally installed fonts. It should also allow us to package fonts as snaps in the near future. The underlying technology is very generic and I'm sure we'll find many interesting things to share this way.
As always, you can reach out to us on IRC (#snappy) and on the mailing list (snapcraft@lists.ubuntu.com, see this post for details). If you have any questions we will be happy to answer them.
See you next week!
snapd (2.0.5) xenial; urgency=medium * New upstream release: LP: #1583085 - interfaces: add dbusmenu, freedesktop and kde notifications to unity7 (LP: #1573188) - daemon: make localSnapInfo return SnapState - cmd: make snap list with no snaps not special - debian: workaround for XDG_DATA_DIRS issues - cmd,po: fix conflicts, apply review from #1154 - snap,store: load and store the private flag sent by the store in SideInfo - interfaces/apparmor/template.go: adjust /dev/shm to be more usable - store: use purchase decorator in Snap and FindSnaps - interfaces: first version of the networkmanager interface - snap, snappy: implement the new (minmimal) kernel spec - cmd/snap, debian: move manpage generation to depend on an environ key; also, fix completion -- Michael VogtThu, 19 May 2016 15:29:16 +0200 snapd (2.0.4) xenial; urgency=medium * New upstream release: - interfaces: cleanup explicit denies - integration-tests: remove the ancient integration daemon tests - integration-tests: add network-bind interface test - integration-tests: add actual checks for undoing install - integration-tests: add store login test - snap: add certain implicit slots only on classic - integration-tests: add coverage flags to snapd.service ExecStart setting when building from branch - integration-tests: remove the tests for features removed in 16.04. - daemon, overlord/snapstate: "(de)activate" is no longer a thing - docs: update meta.md and security.md for current snappy - debian: always start snapd - integration-tests: add test for undoing failed install - overlord: handle ensureNext being in the past - overlord/snapstate,overlord/snapstate/backend,snappy: start backend porting LinkSnap and UnlinkSnap - debian/tests: add reboot capability to autopkgtest and execute snapPersistsSuite - daemon,snappy,progress: drop license agreement broken logic - daemon,client,cmd/snap: nice access denied message (LP: #1574829) - daemon: add user parameter to all commands - snap, store: rework purchase methods into decorators - many: simplify release package and add OnClassic - interfaces: miscellaneous policy updates - snappy,wrappers: move desktop files handling to wrappers - snappy: remove some obviously dead code - interfaces/builtin: quote apparmor label - many: remove the gadget yaml support from snappy - snappy,systemd,wrappers: move service units generation to wrappers - store: add method to determine if a snap must be bought - store: add methods to read purchases from the store - wrappers,snappy: move binary wrapper generation to new package wrappers - snap: add `snap help` command - integration-tests: remove framework-test data and avoid using config-snap for now - builtin/unity7.go: allow using gmenu. Closes: LP:#1576287 - add integration test to verify fix for LP:#1571721 -- Michael Vogt Fri, 13 May 2016 17:19:37 -0700
Friday, May 13, 2016
snapd updated to 2.0.3
Ubuntu 16.04 has just been updated with a new release of snapd (2.0.3)
Our release manager, Michael Vogt, has prepared and pushed this release into the Ubuntu archive. You can look at the associated milestone sru-1 on Launchpad for more details.
Work is already under way on sru-2
You can find the changelog below.
Our release manager, Michael Vogt, has prepared and pushed this release into the Ubuntu archive. You can look at the associated milestone sru-1 on Launchpad for more details.
Work is already under way on sru-2
You can find the changelog below.
* New upstream micro release: - integration-tests, debian/tests: add unity snap autopkg test - snappy: introduce first feature flag for assumes: common-data-dir - timeout,snap: add YAML unmarshal function for timeout.Timeout - many: go into state.Retry state when unmounting a snap fails. (LP: #1571721, #1575399) - daemon,client,cmd/snap: improve output after snap install/refresh/remove (LP: #1574830) - integration-tests, debian/tests: add test for home interface - interfaces,overlord: support unversioned data - interfaces/builtin: improve the bluez interface - cmd: don't include the unit tests when building with go test -c for integration tests - integration-tests: teach some new trick to the fake store, reenable the app refresh test - many: move with some simplifications test snap building to snap/snaptest - asserts: define type for revision related errors - snap/snaptest,daemon,overlord/ifacestate,overlord/snapstate: unify mocking snaps behind MockSnap - snappy: fix openSnapFile's handling of sideInfo - daemon: improve snap sideload form handling - snap: add short and long description to the man-page (LP: #1570280) - snappy: remove unused SetProperty - snappy: use more accurate test data - integration-tests: add a integration test about remove removing all revisions - overlord/snapstate: make "snap remove" remove all revisions of a snap (LP: #1571710) - integration-tests: re-enable a bunch of integration tests - snappy: remove unused dbus code - overlord/ifacestate: fix setup-profiles to use new snap revision for setup (LP: #1572463) - integration-tests: add regression test for auth bug LP:#1571491 - client, snap: remove obsolete TypeCore which was used in the old SystemImage days - integration-tests: add apparmor test - cmd: don't perform type assertion when we know error to be nil - client: list correct snap types - intefaces/builtin: allow getsockname on connected x11 plugs (LP: #1574526) - daemon,overlord/snapstate: read name out of sideloaded snap early, improved change summary - overlord: keep tasks unlinked from a change hidden, prune them - integration-tests: snap list on fresh boot is good again - integration-tests: add partial term to the find test - integration-tests: changed default release to 16 - integration-tests: add regression test for snaps not present after reboot - integration-tests: network interface - integration-tests: add proxy related environment variables to snapd env file - README.md: snappy => snap - etc: trivial typo fix (LP:#1569892) - debian: remove unneeded /var/lib/snapd/apparmor/additional directory (LP: #1569577)
Wednesday, April 27, 2016
Anatomy of a snappy interface
This post is the third in a series about snappy interfaces. Knowledge presented in posts one and two is assumed.
Today we will look at what makes an interface. This post might be a bit heavier on the programming side so feel free to skip over the code fragments if that is not your thing. Note that we will not build an actual interface just yet. The goal of this article is to set the stage for that to be meaningful, at least in part.
In go, this is spelled out as:
This can be read as "the go type Interface is an object with the following methods ..."
Both methods take an object to sanitize (a plug or a slot) and return an error if the object is incorrect. If you don't need to check anything just return nil and carry on.
Each security system is responsible for setting up security for each app of each snap. By default all apps get the same treatment, there is nothing unique about any particular app. Interfaces allow to pass extra information to a particular security system to let a particular app do more than it could otherwise.
This extra information is exchanged as snippets. Snippets are just bits of untyped data, blobs, sequences of bytes. In practice all current snippets are just pieces of text that are easy to read and understand.
Interfaces can hand out snippets for each of the four distinct cases:
Typically most permissions will be based around a plug connected to a slot. Apps bound to the plug will be allowed to talk to a specific socket, to a specific DBus object, to a specific device on the system. All such permissions will be expressed through a snippet provided by case 2 in the list above.
For applications providing services to other snaps (e.g. bluez, network-manager, pulseaudio, mir, X11) the mere fact of having a slot will grant permissions to create the service (to configure network, manage GPUs, etc). Applications like this will use the mechanism provided by case 3 in the list above.
The meaning of the snippets is almost opaque to snappy. Snappy collects them, assembles them together and hands them over to security backends to process. At the end of the day they end up as various configuration files.
So how does the method definition look like? Like this:
This feature was designed to let snappy automatically connect plugs in snaps being installed if there is a viable, unique slot on the OS snap that satisfies the interface requirements. If you recall, the OS snap exposes a number of slots for things like network, network-bind and so on. To make the user experience better, when a snap wants to use one of those interfaces the user does not have to connect them explicitly.
Please note that we're going to be conservative in what can be connected automatically. As a rule of thumb auto-connection is allowed if this is a reasonable thing to do and it is not a serious security risk (the interface doesn't hand out too much power).
How this actually gets used and how the snippets should look like, that is for the next post.
Today we will look at what makes an interface. This post might be a bit heavier on the programming side so feel free to skip over the code fragments if that is not your thing. Note that we will not build an actual interface just yet. The goal of this article is to set the stage for that to be meaningful, at least in part.
The Interface interface
From the point of view of snappy, an Interface is a bit of code with specific APIs. In go's terms it is an interface (note the lower case i). If you are familiar with other languages it is just a way to describe a class with a specific set of methods.In go, this is spelled out as:
type Interface interface { ... }
This can be read as "the go type Interface is an object with the following methods ..."
Interface name
At a very basic level each interface has a name.type Interface interface { Name() string ... }That is, having an arbitrary interface you call the Name method to obtain the name of that interface. Interface name must be unique and is something that other developers will refer to so plan ahead and pick a good, descriptive name. You cannot just change it later.
Validating plugs and slots
Two of the methods in an Interface are used to verify if a plug or slot definition is correct.type Interface interface { ... SanitizePlug(plug *Plug) error SanitizeSlot(slot *Slot) error ... }
Remember that plugs and slots can hold arbitrary attributes. A particular interface, say, one that allows access to a GPIO pin, can use an attribute to describe which particular pin is exposed. As an interface author you should check if the pin is specified correctly (e.g. that it is a number, that it has a sensible value, etc).
Both methods take an object to sanitize (a plug or a slot) and return an error if the object is incorrect. If you don't need to check anything just return nil and carry on.
Interfaces and snippets
Having a valid plug and slot, the main thing that interfaces do is to influence the various security systems. This is implemented as a set of four methods. Before I will spill the beans on the code I will explain this informally.
Small digression, when you see a security system below think of things like apparmor and seccomp. I will focus on security systems in a dedicated instalment. For now they are simply a part of the overall security story.
Each security system is responsible for setting up security for each app of each snap. By default all apps get the same treatment, there is nothing unique about any particular app. Interfaces allow to pass extra information to a particular security system to let a particular app do more than it could otherwise.
This extra information is exchanged as snippets. Snippets are just bits of untyped data, blobs, sequences of bytes. In practice all current snippets are just pieces of text that are easy to read and understand.
Interfaces can hand out snippets for each of the four distinct cases:
- the mere fact of having a plug of a given interface
- the fact of having a particular plug connected to a particular slot
- the mere fact of having a slot of a given interface
- the fact of having a particular slot connected to a particular plug
Typically most permissions will be based around a plug connected to a slot. Apps bound to the plug will be allowed to talk to a specific socket, to a specific DBus object, to a specific device on the system. All such permissions will be expressed through a snippet provided by case 2 in the list above.
For applications providing services to other snaps (e.g. bluez, network-manager, pulseaudio, mir, X11) the mere fact of having a slot will grant permissions to create the service (to configure network, manage GPUs, etc). Applications like this will use the mechanism provided by case 3 in the list above.
The meaning of the snippets is almost opaque to snappy. Snappy collects them, assembles them together and hands them over to security backends to process. At the end of the day they end up as various configuration files.
So how does the method definition look like? Like this:
type Interface interface { ... PermanentPlugSnippet(plug *Plug, securitySystem SecuritySystem) ([]byte, error) ConnectedPlugSnippet(plug *Plug, slot *Slot, securitySystem SecuritySystem) ([]byte, error) PermanentSlotSnippet(slot *Slot, securitySystem SecuritySystem) ([]byte, error) ConnectedSlotSnippet(plug *Plug, slot *Slot, securitySystem SecuritySystem) ([]byte, error) ... }Note that each method gets the name of the security system as an argument. A single interface can influence all security systems if that is required for it to operate correctly.
Connecting interfaces automatically
The one last thing an interface can do is say it wants to automatically connect plugs to viable slots under certain conditions. This is expressed as the following method:type Interface interface { ... AutoConnect() bool }
This feature was designed to let snappy automatically connect plugs in snaps being installed if there is a viable, unique slot on the OS snap that satisfies the interface requirements. If you recall, the OS snap exposes a number of slots for things like network, network-bind and so on. To make the user experience better, when a snap wants to use one of those interfaces the user does not have to connect them explicitly.
Please note that we're going to be conservative in what can be connected automatically. As a rule of thumb auto-connection is allowed if this is a reasonable thing to do and it is not a serious security risk (the interface doesn't hand out too much power).
The complete picture
You can check the complete interface code, with documentation, here. The key thing to take out of this whole article is that interfaces are bits of code that can validate plugs and slots and hand out security snippets.How this actually gets used and how the snippets should look like, that is for the next post.
Friday, April 22, 2016
snap install --devmode
This is very short post about a particular feature of snappy that many people don't know about.
You can install a snap using "snap install --devmode ...". The presence of --devmode switches confinement, for that snap only, from enforcing mode where the application is rejected from accessing certain things (or killed, depending on access pattern) to complain mode where the application is allowed to do anything, as the local user could, but appropriate messages are logged to let developers know that something would not normally work.
Development mode is aimed at snap developers to let them understand what their application (or application toolkit) is doing under the hood. We are working on a separate tool that works along --devmode applications, analysing log files and offering suggestions as to which interfaces to use.
You can install a snap using "snap install --devmode ...". The presence of --devmode switches confinement, for that snap only, from enforcing mode where the application is rejected from accessing certain things (or killed, depending on access pattern) to complain mode where the application is allowed to do anything, as the local user could, but appropriate messages are logged to let developers know that something would not normally work.
Development mode is aimed at snap developers to let them understand what their application (or application toolkit) is doing under the hood. We are working on a separate tool that works along --devmode applications, analysing log files and offering suggestions as to which interfaces to use.
Wednesday, April 20, 2016
Snappy interfaces, plugs, slots and connections
This is the second post in a series about snappy interfaces.
In the previous instalment we looked at the basics of interfaces. We used a snap containing the "links" text mode web browser and the "network" plug to get access to the network. We took advantage of the auto-connection feature of the "network" interface to simply install the snap and have it do the right thing.
Today we will focus on connections. A connection is simply an ordered tuple (plug, slot) where plug refers to a particular plug in a particular snap and slot refers to a particular slot in a particular (typically different) snap. Before we really talk about connections I need to explore plugs and slots with a little more depth.
Recall the snapcraft.yaml file we looked at the last time:
Let's focus on just the plug syntax:
What you see above is an abbreviated form. Snappy has many abbreviated forms that make it look nice and are better as long as you understand what they mean. The fully expanded form, having the exact same semantics, looks like this:
Note that plug name is independent from plug interface name. As a convenience, when the plug name is the same as the interface name the mapping of plug name to interface name can be left out. As another convenience if the plug doesn't require any additional attributes (more on that later) the whole top-level declaration is optional. This is why we could get away with just plugs: [network, network-bind] defined in the app level.
The same is true of slots. Virtually every time we talk about plugs the exact same thing is true for slots. I will not mention that again unless the rule is broken.
Let's recap some of the key facts:
Snappy maintains a persistent internal data structure called the state. This is used to record connections as they are established. When you reboot your snappy device the state is used to re-load all connections. Right now there is no difference but once we add support for hooks things will get more interesting. On each reboot the details of particular plugs or slots can be different. For example a hypothetical cheese snap can be connected to the same physical USB camera but due to way USB enumeration works some attributes of the slot that represents that camera might differ (e.g. the particular device that cheese can access can have a different path on the filesystem).
When connections are established the various systems that maintain snappy's security model must be informed. Each security system uses information about plugs, slots, apps, and connections to come up with a set of rules or profiles that encode what each application is or is not allowed to do. I will explore profiles in depth later, for now just remember that profiles are specific to an application. A given snap can have different security profiles for each of its applications based on how plugs and slots are bound to those applications.
Note that only the links app refers to plugs, the bookmarks app does not. If a plug or slot is declared in a snap, but not associated with a specific application they will be implicitly bound to all apps. When a plug or slot is specified by one or more apps, as in the above example, it will be bound only to those applications. Compare that to the following code:
Here both network and network-bind plugs are not referred from any application explicitly and thus they implicitly bind to all apps. Keep this in mind, the difference is subtle but the effective security is not.
Remember: the order for all of those operations is "snap:plug snap:slot"
$ sudo snap disconnect links:network ubuntu-core:network
$ sudo snap disconnect links:network-bind ubuntu-core:network-bind
After running the two commands links won't be able to use the system calls required for accessing the network and will effectively be offline.
At time of writing, links would not even start correctly, as it would be killed in the attempt to use particular network-related system calls, but I suspect that over time we will tweak the interfaces system to just put apps into an offline sandbox as if you had yanked the network cables and put enough tinfoil around your device to shield it from all signals. I would certainly love to be able to start an arbitrary app without letting it talk to the network.
If you inspect the output of "snap interfaces" you will see that it has changed slightly, the relevant parts now look like this:
This output tells us that both plugs and slots are disconnected. To restore links to an usable state you can re-connect it with the following two commands:
$ sudo snap connect links:network ubuntu-core:network
$ sudo snap connect links:network-bind ubuntu-core:network-bind
That's it for today. Next time we will take a closer look at security systems and how they interact with interfaces internally. This will allow us to explore creating new interfaces for things that don't have one yet.
As before please feel free to post your comment below. I will gladly answer all questions related to the interface system.
Special thanks to Jonathan Cave for proofreading and improvements!
In the previous instalment we looked at the basics of interfaces. We used a snap containing the "links" text mode web browser and the "network" plug to get access to the network. We took advantage of the auto-connection feature of the "network" interface to simply install the snap and have it do the right thing.
Today we will focus on connections. A connection is simply an ordered tuple (plug, slot) where plug refers to a particular plug in a particular snap and slot refers to a particular slot in a particular (typically different) snap. Before we really talk about connections I need to explore plugs and slots with a little more depth.
Plugs and slots
As the most essential fact, both plugs and slots have a name and an interface name. In the examples so far this was somewhat shrouded because the plug name and plug interface name had the same value. Let's take all the simplifications away and look at the raw facts.Recall the snapcraft.yaml file we looked at the last time:
name: links version: 2.12-1 summary: Web browser running in text mode description: | Links is a text mode WWW browser, similar to Lynx. It displays tables, frames, downloads on background, uses HTTP/1.1 keepalive connections. apps: links: command: links plugs: [network, network-bind] parts: links: plugin: nil stage-packages: - links
Let's focus on just the plug syntax:
apps: links: command: links plugs: [network, network-bind]
What you see above is an abbreviated form. Snappy has many abbreviated forms that make it look nice and are better as long as you understand what they mean. The fully expanded form, having the exact same semantics, looks like this:
apps: links: command: links plugs: [network, network-bind] plugs: network: interface: network network-bind: interface: network-bind
Note that plug name is independent from plug interface name. As a convenience, when the plug name is the same as the interface name the mapping of plug name to interface name can be left out. As another convenience if the plug doesn't require any additional attributes (more on that later) the whole top-level declaration is optional. This is why we could get away with just plugs: [network, network-bind] defined in the app level.
The same is true of slots. Virtually every time we talk about plugs the exact same thing is true for slots. I will not mention that again unless the rule is broken.
Let's recap some of the key facts:
- Plugs and slots are objects that have several attributes
- Name uniquely identifies a plug or a slot within a particular snap.
- Interface name describes the "kind" of plug/slot.
Connections
So, returning to connections. A connection can be made between a plug and a slot of the same interface type. For example snappy would not allow a connection between the unity7 plug and the network slot. Snappy doesn't allow this. On IOT-like devices you can expect slots using names very different from the interface type they represent. Some made-up examples are "user-button", "led5" or "gpio-12". At the same time you can expect more manual, explicit connections to be established on the command line or through other mechanisms as there will be many possible candidates for a particular slot to form a connection.Snappy maintains a persistent internal data structure called the state. This is used to record connections as they are established. When you reboot your snappy device the state is used to re-load all connections. Right now there is no difference but once we add support for hooks things will get more interesting. On each reboot the details of particular plugs or slots can be different. For example a hypothetical cheese snap can be connected to the same physical USB camera but due to way USB enumeration works some attributes of the slot that represents that camera might differ (e.g. the particular device that cheese can access can have a different path on the filesystem).
When connections are established the various systems that maintain snappy's security model must be informed. Each security system uses information about plugs, slots, apps, and connections to come up with a set of rules or profiles that encode what each application is or is not allowed to do. I will explore profiles in depth later, for now just remember that profiles are specific to an application. A given snap can have different security profiles for each of its applications based on how plugs and slots are bound to those applications.
Binding plugs and slots to app
So what does a binding look like? Let's say that links grew a bookmark manager app that can be started separately from links itself. Let's also assume that, for tighter security, we will not allow that application to talk to the network. This is how this could be encoded in snapcraft.yaml:... apps: links: command: links plugs: [network, network-bind] bookmarks: command: links-bookmark-editor (fake) ...
Note that only the links app refers to plugs, the bookmarks app does not. If a plug or slot is declared in a snap, but not associated with a specific application they will be implicitly bound to all apps. When a plug or slot is specified by one or more apps, as in the above example, it will be bound only to those applications. Compare that to the following code:
... apps: links: command: links bookmarks: command: links-bookmark-editor (fake) plugs: network: interface: network network-bind: interface: network-bind ...
Here both network and network-bind plugs are not referred from any application explicitly and thus they implicitly bind to all apps. Keep this in mind, the difference is subtle but the effective security is not.
Working with connections
Snappy allows users to change connections using two command line commands: snap connect and snap disconnect. Both of those respond to --help so make sure to read that if you are exploring.Caveat: as of snapd 2.0.2 connect/disconnect only accept the fully-spelled-out form "snap connect snap:plug snap:slot". Other variants don't work yet, please avoid them for now.For example, if you installed links as a snap earlier, you could now disconnect it from the network and network-bind slots and see what happens. Let's do so now:
Remember: the order for all of those operations is "snap:plug snap:slot"
$ sudo snap disconnect links:network ubuntu-core:network
$ sudo snap disconnect links:network-bind ubuntu-core:network-bind
After running the two commands links won't be able to use the system calls required for accessing the network and will effectively be offline.
At time of writing, links would not even start correctly, as it would be killed in the attempt to use particular network-related system calls, but I suspect that over time we will tweak the interfaces system to just put apps into an offline sandbox as if you had yanked the network cables and put enough tinfoil around your device to shield it from all signals. I would certainly love to be able to start an arbitrary app without letting it talk to the network.
If you inspect the output of "snap interfaces" you will see that it has changed slightly, the relevant parts now look like this:
Slot Plug :network - :network-bind - - links:network - links:network-bind
This output tells us that both plugs and slots are disconnected. To restore links to an usable state you can re-connect it with the following two commands:
$ sudo snap connect links:network ubuntu-core:network
$ sudo snap connect links:network-bind ubuntu-core:network-bind
That's it for today. Next time we will take a closer look at security systems and how they interact with interfaces internally. This will allow us to explore creating new interfaces for things that don't have one yet.
As before please feel free to post your comment below. I will gladly answer all questions related to the interface system.
Special thanks to Jonathan Cave for proofreading and improvements!
Snappy, snapcraft and interfaces
This is the first of a series of articles about snappy, specifically focusing on snappy interfaces.
Interfaces are an essential part of snappy. They are the mechanism which allows snaps to reach beyond the default space they are given, to interact with other parts of the system, with other snaps, with the network, with the desktop and more.
Before we begin we should define some terminology. I will do my best to use the right terms each time. During the development of snappy 2.0 series we tried various different terms for what is now known as snappy interfaces. If you've been following the development release for the past few months you may still remember some of the older terms and that can add to the confusion as to what interfaces are. Let's set this straight now:
Snap is the new package format. Snaps hold one or more applications.
App is a runnable executable exposed by a snap. There are command line apps, desktop apps and background apps (aka daemon/service).
Interface is the mechanism with which two snaps can interact with each other. The type of interaction is defined by the interface name. Interfaces can allow snaps to communicate, share particular resources or access particular hardware.
Plug is the consumer part of an interface. Plugs can be connected to slots. Most snaps will define one or more plugs.
Slot is the producer or provider part of an interface. Slots are what plugs connect to. Many slots come directly from the OS snap. Most snaps will not define any slots.
Don't worry if the interface/plug/slot part is still confusing. Let's explore a single example today and see how plugs and slots play a part of the experience. For this purpose I've created a snap with the links text mode web browser.
The entire snapcraft.yaml file for this snap is reproduced below:
If you are not familiar with snapcraft I would recommend taking a quick look at the introduction to snapcraft. Even if you haven't used snapcraft yet the content is somewhat self-explanatory. The snap "links" contains one app, "links", which runs the command "links". The snap is comprised of just one part, "links", which is using the "nil" (do nothing extra) plugin to put the "links" Debian package into the stage directory from which our snap is created.
All that this does it puts pre-packaged links executable into a snap. No compiling required!
In Ubuntu, Debian and many other distributions, applications started by a particular user run with the privileges of that user. In other words *every* application that you install from outside of the official repository has complete and unrestricted access to all of your personal files. To your photos. To your browser history. To your saved passwords. To your ssh and gpg keys. Each application you ever installed from a PPA also has complete access to your kernel and to all the other applications. Wow, that's a lot of responsibility. This is why in Ubuntu the trust is put on the Debian / Ubuntu repository. Trustworthy people review the software that is placed there to ensure it doesn't have any nasty code inside.
So why did we have use the two plugs in the snapcraft.yaml above? Because snappy is different. Similarly to how your phone may work, snaps don't get unrestricted access to all of your hardware and personal data.
Recall that snappy runs all apps in a confined sandbox. This is the same technology we use on the Ubuntu phone and tablet. I won't go into details as to how this snadbox works but suffice it to say that apps don't have access to the network or to your personal files default. To give links access to the internet we have to use snappy interfaces.
So what are those interfaces? The network interface allows applications to access the network as a client. This is clearly what links wants to do, it wants to have access to the internet. The network-bind interface allows applications to access the network as a server. Why does links need this? Because apparently it has an option to bind to particular local IP address. It's somewhat unfortunate but we cannot avoid it without patching this feature out of links. Let's not do that today. This is exactly what the statement "plugs: [network, network-bind]" above does.
Please recall that interfaces have two sides, the plug side and the slot side. The plugs are meant for consumer. The slots are meant for producers (or providers). So who's providing network and network-bind on our system? The OS does.
The OS snap, ubuntu-core, has a slot of type network (many plugs will be named after the interface name but those names are independent). When the snap links was installed snappy automatically connected the plug network from the snap links to the slot network from the snap ubuntu-core. The same thing happened for network-bind. Both the network and the network-bind interfaces are designed to automatically-connect to the slots on the OS snap.
You can see that this is true by running the command "snap interfaces" which shows all plugs, slots and their connections.
Let's have a look at the output. There are two columns, one for slots and one for plugs. In general each plug and slot is displayed as snap:plug or snap:slot but as you can see above there are a number of abbreviations that make the output less repetitive. The Slot column doesn't display the name of the OS snap (ubuntu-core). The Plug column doesn't display the plug name if it is the same as interface name (here it just displays the snap name twice).
Had both of those abbreviations been turned off the relevant portion of the output would look something like this:
You can install this example snap on your system (make sure to run up-to-date Ubuntu 16.04 on an amd64 architecture) by running "sudo snap install links". After that it should just work, you should be able to run links and browse the web in all of its plain-text glory.
In the next article I will explore connections. Show you how you can disconnect, connect and what happens when you do that.
If you have any questions please leave them as comments below.
Interfaces are an essential part of snappy. They are the mechanism which allows snaps to reach beyond the default space they are given, to interact with other parts of the system, with other snaps, with the network, with the desktop and more.
Before we begin we should define some terminology. I will do my best to use the right terms each time. During the development of snappy 2.0 series we tried various different terms for what is now known as snappy interfaces. If you've been following the development release for the past few months you may still remember some of the older terms and that can add to the confusion as to what interfaces are. Let's set this straight now:
Snap is the new package format. Snaps hold one or more applications.
App is a runnable executable exposed by a snap. There are command line apps, desktop apps and background apps (aka daemon/service).
Interface is the mechanism with which two snaps can interact with each other. The type of interaction is defined by the interface name. Interfaces can allow snaps to communicate, share particular resources or access particular hardware.
Plug is the consumer part of an interface. Plugs can be connected to slots. Most snaps will define one or more plugs.
Slot is the producer or provider part of an interface. Slots are what plugs connect to. Many slots come directly from the OS snap. Most snaps will not define any slots.
Don't worry if the interface/plug/slot part is still confusing. Let's explore a single example today and see how plugs and slots play a part of the experience. For this purpose I've created a snap with the links text mode web browser.
The entire snapcraft.yaml file for this snap is reproduced below:
name: links version: 2.12-1 summary: Web browser running in text mode description: | Links is a text mode WWW browser, similar to Lynx. It displays tables, frames, downloads on background, uses HTTP/1.1 keepalive connections. apps: links: command: links plugs: [network, network-bind] parts: links: plugin: nil stage-packages: - links
If you are not familiar with snapcraft I would recommend taking a quick look at the introduction to snapcraft. Even if you haven't used snapcraft yet the content is somewhat self-explanatory. The snap "links" contains one app, "links", which runs the command "links". The snap is comprised of just one part, "links", which is using the "nil" (do nothing extra) plugin to put the "links" Debian package into the stage directory from which our snap is created.
All that this does it puts pre-packaged links executable into a snap. No compiling required!
In Ubuntu, Debian and many other distributions, applications started by a particular user run with the privileges of that user. In other words *every* application that you install from outside of the official repository has complete and unrestricted access to all of your personal files. To your photos. To your browser history. To your saved passwords. To your ssh and gpg keys. Each application you ever installed from a PPA also has complete access to your kernel and to all the other applications. Wow, that's a lot of responsibility. This is why in Ubuntu the trust is put on the Debian / Ubuntu repository. Trustworthy people review the software that is placed there to ensure it doesn't have any nasty code inside.
So why did we have use the two plugs in the snapcraft.yaml above? Because snappy is different. Similarly to how your phone may work, snaps don't get unrestricted access to all of your hardware and personal data.
Recall that snappy runs all apps in a confined sandbox. This is the same technology we use on the Ubuntu phone and tablet. I won't go into details as to how this snadbox works but suffice it to say that apps don't have access to the network or to your personal files default. To give links access to the internet we have to use snappy interfaces.
So what are those interfaces? The network interface allows applications to access the network as a client. This is clearly what links wants to do, it wants to have access to the internet. The network-bind interface allows applications to access the network as a server. Why does links need this? Because apparently it has an option to bind to particular local IP address. It's somewhat unfortunate but we cannot avoid it without patching this feature out of links. Let's not do that today. This is exactly what the statement "plugs: [network, network-bind]" above does.
Please recall that interfaces have two sides, the plug side and the slot side. The plugs are meant for consumer. The slots are meant for producers (or providers). So who's providing network and network-bind on our system? The OS does.
The OS snap, ubuntu-core, has a slot of type network (many plugs will be named after the interface name but those names are independent). When the snap links was installed snappy automatically connected the plug network from the snap links to the slot network from the snap ubuntu-core. The same thing happened for network-bind. Both the network and the network-bind interfaces are designed to automatically-connect to the slots on the OS snap.
You can see that this is true by running the command "snap interfaces" which shows all plugs, slots and their connections.
zyga@zyga-thinkpad-w510:~$ snap interfaces Slot Plug :firewall-control - :home - :locale-control - :log-observe - :mount-observe - :network links :network-bind links :network-control - :network-observe - :opengl - :snapd-control - :system-observe - :timeserver-control - :timezone-control - :unity7 - :x11 -
Let's have a look at the output. There are two columns, one for slots and one for plugs. In general each plug and slot is displayed as snap:plug or snap:slot but as you can see above there are a number of abbreviations that make the output less repetitive. The Slot column doesn't display the name of the OS snap (ubuntu-core). The Plug column doesn't display the plug name if it is the same as interface name (here it just displays the snap name twice).
Had both of those abbreviations been turned off the relevant portion of the output would look something like this:
zyga@zyga-thinkpad-w510:~$ snap interfaces Slot Plug ubuntu-core:network links:network ubuntu-core:network-bind links:network-bind
You can install this example snap on your system (make sure to run up-to-date Ubuntu 16.04 on an amd64 architecture) by running "sudo snap install links". After that it should just work, you should be able to run links and browse the web in all of its plain-text glory.
In the next article I will explore connections. Show you how you can disconnect, connect and what happens when you do that.
If you have any questions please leave them as comments below.
Subscribe to:
Posts (Atom)