Debugging snaps
When debugging or tracking potential issues with snaps running within a confined environment, the following approaches, techniques and insights can help resolve issues quickly:
- Use snap try to install a snap from its unpacked and modifiable directory of components
-
Use
snap run --shell
to inpect and test the confined environment. - Developer mode, allows policy violations to proceed to help isolate runtime errors
- Debug logs, track, monitor and check for policy violations
- snap-seccomp and re-exec, resolving seccomp version mismatches under re-execution
- snappy-debug, a snap developed for debugging and policy introspection
- File permissions, along with cgroup device access, can inadvertently limit a snap’s functionality
In addition to the above, GDB can also be used from within a snap’s environment to help isolate and identify potential issues. See Using gdb and gsbserver for more details.
For details on how AppArmor, Seccomp and device permission security policies are implemented, see Security policy and sandboxing.
Developer mode
To help isolate runtime errors when building and testing a snap, a snap can be installed using developer mode.
To install a snap in developer mode, use the --devmode
argument:
$ sudo snap install --devmode mysnap
When a snap is installed with developer mode, violations against a snap’s security policy are permitted to proceed but logged via journald.
Run a shell in the confined environment
To investigate and test the confined environment of a snap, you can open a bash
shell in it. After the snap is installed, use the --shell <name>.<command>
argument of snap run
:
$ snap run --shell mysnap.mycommand
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
This will create the confined environment of the Snap, execute the command-chain and then run bash
inside that environment.
You can then investigate which files your snap has access to by running commands such as ls
and cat
.
It’s important to put --shell
before the name of the snap. Otherwise it will be interpreted as an argument to the application instead of an argument to snap run
.
Debug logs
The easiest way to check for snap policy violations is to search the logs for audit:
$ sudo journalctl --since=yesterday | grep audit
The above command uses --since=yesterday
to limit the typically verbose logging output from journalctl.
A handy debugging technique is to tail/follow journalctl output while exercising the snap:
$ sudo sysctl -w kernel.printk_ratelimit=0 ; journalctl --follow | grep audit
As shown above, kernel log rate limiting can be disabled manually with:
$ sudo sysctl -w kernel.printk_ratelimit=0
If you believe there is a bug in a security policy or want to request and/or contribute a new interface, please file a bug, adding the snapd-interface
tag, and feel free to discuss policy issues on the forum.
AppArmor violations
An AppArmor violation will look something like the following and include apparmor=DENIED
:
audit: type=1400 audit(1431384420.408:319): apparmor="DENIED" operation="mkdir" profile="snap.foo.bar" name="/var/lib/foo" pid=637 comm="bar" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
If there are no AppArmor denials, AppArmor shouldn’t be blocking the snap.
To better understand AppArmor policy for a strictly installed snap, modify the AppArmor policy in place on the target system. Changes aren’t persistent, but this can help when considering a snapd patch or bug report.
For example:
- build the snap
- copy the snap to the target device and install it (or use snap try)
- use the snap (perhaps using
snap run --shell <name>.<command>
), monitoring via journalctl for denials - modifying
/var/lib/snapd/apparmor/profiles/snap.<name>.<command>
as needed (eg, adding rules before the final'}'
)and runningsudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.<name>.<command>
to compile and load the policy into the kernel - use
sudo service snap.<name>.<command> stop/start/etc
as needed for daemons - repeat until AppArmor policy issues are resolved
Seccomp violations
A seccomp violation will look something like:
audit: type=1326 audit(1430766107.122:16): auid=1000 uid=1000 gid=1000 ses=15 pid=1491 comm="env" exe="/bin/bash" sig=31 arch=40000028 syscall=983045 compat=0 ip=0xb6fb0bd6 code=0x0
The syscall=983045
can be resolved by running the scmp_sys_resolver
command on a system of the same architecture as the one with the seccomp violation:
$ scmp_sys_resolver 983045
set_tls
If there are no seccomp violations, seccomp isn’t blocking the snap.
If you notice compat=1
in the seccomp denial, then specify the correct compatibility architecture to scmp_sys_resolver
with -a <arch>
. Eg, if on an amd64 system, use scmp_sys_resolver -a x86 191
(use -a arm
on arm64 systems).
The seccomp filter profile in expected to be located in /var/lib/snapd/seccomp/bpf/*.src (formerly /var/lib/snapd/seccomp/profiles).
The seccomp profile source (the .src file in the profile directory) needs to be recompiled into the profile binary (.bin in the profile directory) as follows:
$ sudo /usr/lib/snapd/snap-seccomp compile /var/lib/snapd/seccomp/bpf/snap.$SNAP_NAME.src /var/lib/snapd/seccomp/bpf/snap.$SNAP_NAME.bin
The snap-confine
command will load the bpf in the .bin
file for the command when you (re)launch the command or snap run --shell
. The seccomp policy language is considerably simpler and is essentially a list of allowed syscalls.
When done, copy any changes you make to /var/lib/snapd/apparmor/profiles/snap.<name>.<command>
or /var/lib/snapd/seccomp/bpf/snap.<name>.<command>.src
to your interface code.
snap-seccomp versions and paths
Tools such as snap-confine, snap-seccomp and snap-exec are internal to snapd and are initially installed with a distribution’s snapd package.
On certain distributions, these tools can become superseded by versions embedded in subsequently installed core and snapd snaps. When developing a seccomp profile, it is important that the correct snap-seccomp binary is used. This can be determined by inspecting which binary is running as snapd.
With re-execution from the subsequently installed core and snapd snaps, these tools get called using their full path from the same location as the currently running binary. This is visible from /proc
:
# with reexecution
$ sudo ls -l /proc/$(pidof snapd)/exe
lrwxrwxrwx 1 root root 0 Jun 5 10:10 /proc/1994/exe -> /snap/snapd/7777/usr/lib/snapd/snapd
Thus tools such as snap-seccomp will be called using its full path, /snap/snapd/7777/usr/lib/snapd/snap-seccomp
.
Without re-execution, the snapd process is using a binary located in the host filesystem:
# no reexecution
$ sudo ls -l /proc/$(pidof snapd)/exe
lrwxrwxrwx 1 root root 0 06-05 12:49 /proc/808335/exe -> /usr/lib/snapd/snapd
Correspondingly, snap-seccomp will be called using its full path /usr/lib/snapd/snapd
.
The snappy-debug snap
The snappy-debug
snap can be used to help with policy violations:
- it disables kernel log rate limiting so that policy denials are easier to spot
- resolves syscall names while considering
compat
- makes recommendations on how to fix violations
The following command will watch /var/log/syslog (output is initially empty until there’s a policy violation):
$ sudo snappy-debug
INFO: Following '/var/log/syslog'. If have dropped messages, use:
INFO: $ sudo journalctl --output=short --follow --all | sudo snappy-debug
^C
$
It’s recommended that snappy-debug is run alongside journalctl:
$ sudo journalctl --output=short --follow --all | sudo snappy-debug
See snappy-debug --help
for further details.
File permissions
While tradition file permissions are respected and enforced, any violations are not currently logged. Similarly, device cgroups may also block access without logging denials.
To check whether device cgroups are affecting a snap’s device access:
- see if there are any snapd-generated udev rules in
/etc/udev/rules.d/70-snap.$SNAPNAME.rules
- if rules are defined, use
udevadm info /dev/$DEVICE
to see if the snap shows up in TAGS, or see if the/run/udev/tags/snap_$SNAPNAME_$COMMAND
directory exists - examine if the
/sys/fs/cgroup/snap.$SNAPNAME.$COMMAND
directory exists and if the device is listed in/sys/fs/cgroup/devices/snap.$SNAPNAME.$COMMAND/devices.allow
(eg,/dev/kmsg
would have ‘c 1:11 rwm
’ since/dev/kmsg
is a character device with MAJOR:MINOR as 1:11 (seels -l /dev/kmsg
))
For device cgroups, create or modify /etc/udev/rules.d/70-snap.$SNAPNAME.rules
as necessary (eg, KERNEL=="kmsg" TAGS+="snap_$YOURSNAPNAME_$YOURCOMMAND"
would tag /dev/kmsg
for your snap), then run sudo udevadm trigger --action=change
. To undo the access, remove the file and run the udevadm
command again. When done, update the interfaces code based on your changes.
If you believe there is a bug in the security policy or want to request and/or contribute a new interface, please file a bug, adding the snapd-interface
tag.
Further reading
- https://github.com/snapcore/snapd/tree/master/interfaces for existing interface code and policy
- http://manpages.ubuntu.com/manpages/xenial/man5/apparmor.d.5.html
-
http://wiki.apparmor.net/index.php/Profiling_by_hand (but use the paths listed above and don’t use the
aa-genprof
oraa-logprof
tools because they are not yet snappy-aware) - https://github.com/snapcore/snapd/wiki/snap-confine-Overview
- https://assets.ubuntu.com/v1/66fcd858-ubuntu-core-security-whitepaper.pdf
- https://github.com/snapcore/snapd/wiki/Snap-Execution-Environment
- Stracing snap commands
Last updated 4 months ago.